LLM inference serving: hosted APIs vs self-hosted models
Hosted APIs are the fastest way to ship and usually the cheapest at low or uneven volume. Managed endpoints let you run open-weight or fine-tuned models without operating GPUs yourself. Self-hosting with an inference server like vLLM or SGLang gives you control, privacy and predictable cost at high, steady volume, and it needs a team that can run GPU infrastructure, so run the numbers with your real traffic before deciding.
Three ways to serve a model
| Option | You manage | Good when |
|---|---|---|
| Hosted API (OpenAI, Anthropic, Google and others) | Nothing below the API | Speed to market, uneven traffic, frontier quality |
| Managed endpoint (cloud AI platforms, inference providers) | Model choice, scaling settings | Open or fine-tuned models, no GPU operations team |
| Self-hosted inference server | GPUs, scaling, upgrades, security | Data can't leave, high steady volume, custom models |
Hosted APIs
You send a request and pay per token. No servers, no GPUs, the newest models, and scale handled for you.
The trade-offs:
- Your data goes to the provider. Check data retention and training policies, and regional options for regulated data.
- You don't control model versions. Providers update and retire models on their schedule.
- Cost grows with every token, and at high volume it can exceed the cost of your own hardware.
- Rate limits and outages are outside your control.
Managed endpoints
Cloud platforms and inference providers can host an open-weight or fine-tuned model for you, behind an API. You choose the model and the scaling settings, and they run the GPUs. It's a middle path: more control over the model, much less operations work than self-hosting.
Self-hosting
You run an inference server on your own GPUs, in your cloud account or on-premises.
Common inference servers:
- vLLM: open source, high throughput through PagedAttention and continuous batching, an OpenAI-compatible API, and support for LoRA adapters.
- SGLang: open source, fast, with strong support for structured outputs and prefix caching.
- NVIDIA TensorRT-LLM and Triton: highly optimised for NVIDIA GPUs.
- llama.cpp and Ollama: for laptops, CPUs and small servers. See run LLMs locally.
Hugging Face's Text Generation Inference is now in maintenance mode, and its documentation recommends vLLM, SGLang or local engines like llama.cpp going forward.
What self-hosting needs:
- GPUs sized for the model plus the KV cache at your expected load.
- Autoscaling, which is harder than for normal services because GPU instances are slow to start.
- Monitoring of GPU utilisation, queue length, time to first token and throughput.
- Security, patching and model upgrades, by your team.
Is self-hosting cheaper?
Self-hosting is mostly a privacy, control or very-high-volume decision, and only sometimes a cost decision. The hidden cost is idle GPUs: a GPU you pay for by the hour costs the same whether it serves 10 requests or 10,000.
def api_monthly(requests: int, in_tokens: int, out_tokens: int, in_price: float, out_price: float) -> float:
return requests * (in_tokens * in_price + out_tokens * out_price) / 1_000_000
def self_hosted_monthly(gpus: int, gpu_hourly: float, engineer_share: float, engineer_monthly: float) -> float:
return gpus * gpu_hourly * 24 * 30 + engineer_share * engineer_monthly
for monthly_requests in (100_000, 1_000_000, 10_000_000):
api = api_monthly(monthly_requests, 1500, 300, in_price=0.40, out_price=1.60)
own = self_hosted_monthly(gpus=2, gpu_hourly=2.50, engineer_share=0.25, engineer_monthly=12_000)
print(f"{monthly_requests:>10,} requests/month API ${api:>9,.0f} self-hosted ${own:>9,.0f}")
The prices in this example are placeholders, so use your provider's current prices and your real token counts. Also check whether two GPUs can serve your peak load at your latency target, because the self-hosted line only holds if they can.
How to decide
- Estimate monthly tokens from your logs or a realistic forecast.
- Price them on the API you'd use.
- Price self-hosting at that volume: GPUs at peak capacity, engineer time, monitoring and maintenance.
- Add the constraints that aren't cost: data residency, compliance, latency, the models you need.
- Revisit every six months, because prices and models change fast.
Many teams end up with a hybrid: hosted APIs for general and complex requests, a self-hosted model for sensitive or high-volume narrow tasks. See the hybrid fleet pattern.
From the floor
In my webinar for leaders on cloud APIs vs open-source LLMs, the decision came down to three questions: is the API bill becoming a problem at your volume, is data security or compliance a constraint, and do you need performance on your own data that a general model can't give you. If any of these is true, open-source models you host yourself become worth the operations work, and if none is, the API is usually the better choice.
Related
- Cloud APIs vs open-source LLMs
- vLLM vs SGLang vs Ollama
- Hosted vs self-hosted LLMs
- Inference optimization