Rollouts, canaries and model upgrades
Never switch all traffic to a new model or prompt at once. Run the full eval suite first, then shadow mode (process real traffic without showing results), then a small canary share of users, and compare quality, latency, cost and complaints, expanding only if the gates pass. Pin model versions in config, keep the old version ready for instant rollback, and plan upgrades before providers retire the versions you use.
A model upgrade is a migration
A provider releases a new model version, you switch, and your carefully tuned prompts behave differently overnight: answers get longer, a format changes, a refusal appears where there wasn't one, a tool is called in a different order. The new model may be better on average and worse on exactly the cases your product depends on.
Treat every model change, and every significant prompt change, as a migration with a plan.
Pin versions
Use exact model versions in config (dated snapshots where the provider offers them), not aliases that move when the provider updates them. Record the model version on every trace, so you always know which version produced an answer.
Rollout steps
- Offline: run the full eval suite on the new version, and compare per failure mode, not only overall.
- Shadow mode: send a sample of live traffic to the new version as well, don't show its answers, and compare its outputs with judges and spot checks.
- Canary: send 5% of users to the new version. Watch quality signals, latency, cost and complaints.
- Expand: 25%, then 50%, then 100%, with gates at each step.
- Keep the old version available for instant rollback until the new one has run for a full week.
Gates
Write the gates down before the rollout:
| Gate | Example |
|---|---|
| Regression evals | No drop on the regression suite |
| Capability evals | No drop larger than 2 points on any failure mode |
| Latency | p95 time to first token within 10% of the current version |
| Cost | Cost per successful task within budget |
| Online signals | Thumbs down and escalation rates not higher than the current version |
| Errors | Error and fallback rates not higher |
Rollback
Rollback should be one step: a config change or a feature flag, not a redeploy. Test it before you need it.
Provider deprecations
Providers retire older models on a schedule, usually with notice. Track the deprecation dates of every model you use, and start the upgrade weeks before, because the rollout steps above take time. For self-hosted and fine-tuned models, the same applies when you change the base model: adapters have to be retrained and re-evaluated. See serving adapters.
Prompt changes
Prompt changes are smaller than model changes, and they still change outputs. Run them through CI evals, and for important prompts, a canary as well.
Measuring it
- Number of upgrades rolled back.
- Incidents caused by upgrades.
- Time from a new model's release to a decision about it.
From the floor
In the leaders webinar, I described routing requests between a powerful API, a cheap open-source model and a fine-tuned model, and the same router is what makes a gradual rollout possible: send a small share of requests to the new model, compare, and move the rest only when the results hold.