Hosted LLM APIs vs self-hosted models: engineering comparison
Hosted APIs give you the strongest models with no infrastructure, at the cost of provider rate limits, model changes you don't control and data leaving your network. Self-hosting gives you control over versions, data and latency tuning, at the cost of GPU capacity planning, serving engines, scaling, upgrades and security work. For the business decision, see cloud APIs vs open-source LLMs.
Comparison
| Engineering concern | Hosted API | Self-hosted |
|---|---|---|
| Time to first call | Minutes | Days to weeks |
| Model quality ceiling | Highest general models | Best open-weight models |
| Latency control | Limited (region, model choice) | Full (hardware, batching, quantization) |
| Scaling | Provider limits | Your GPU capacity |
| Version control | Provider deprecations | You pin and upgrade |
| Data location | Provider regions | Your network |
| Observability | Request-level | Full stack |
| Team needed | App engineers | Plus ML infrastructure engineers |
What self-hosting adds to your work
- Capacity planning from model size, KV cache and concurrency.
- A serving engine, tuned and monitored. See vLLM vs SGLang vs Ollama.
- Autoscaling, which is slow for GPUs because models take time to load.
- Model upgrades, evaluated before switching.
- Security: patching, access control, and a supply chain for model files.
What hosting adds to your work
- Rate limit handling, retries and fallbacks to another provider.
- Watching for deprecations and re-running evals when a version changes.
- Data terms and region checks for every new use case.
When to choose which
Start hosted unless privacy or data residency rules it out. Move specific high-volume or sensitive workloads to self-hosted models when the numbers or the rules justify it, and keep both behind one gateway so the move is a routing change.
From the floor
In the leaders session on cloud APIs vs open-source LLMs, the hybrid fleet I drew put both behind a model router: an external API for routine questions and a self-hosted, fine-tuned model for complex or sensitive ones, so the engineering choice isn't one or the other for the whole company.
Related
- Cloud APIs vs open-source LLMs
- Inference serving
- vLLM vs SGLang vs Ollama
- Capacity and cost modelling