Fallback chain pattern for LLM systems
For each important LLM call, define an ordered chain of what happens when it fails: retry with backoff, then an alternate model from the same provider, then a different provider, then a cached or precomputed answer, then a simpler non-LLM response, then a handoff to a person. Design what the user sees at each level, because a fallback is part of the product, not only a technical detail. Evaluate every link in the chain on your eval set, because a fallback that answers badly is an outage nobody notices.
Why a chain, not a single fallback
When the primary model fails, many systems either show an error or silently switch to a model that was never tested on the task. Failures come in different kinds (a short timeout, a rate limit, a provider outage, invalid output), and each needs a different response. A chain handles them in order, from the cheapest fix to the most expensive.
Chain
| Level | Trigger | User sees |
|---|---|---|
| 1. Retry | Transient error, timeout | Nothing |
| 2. Alternate model, same provider | Repeated errors, rate limit on one model | Normal answer, maybe slower |
| 3. Alternate provider | Provider outage | Normal answer |
| 4. Cache | All models down, cacheable question | A stored answer |
| 5. Simple response | No model available | A help article, a form, search results |
| 6. Person | High-value or failed tasks | "A member of our team will get back to you" |
Not every call needs every level. A background summarisation job can simply wait and retry later. A customer-facing chat needs the full chain.
Rules for the chain
- Respect data rules: a request that must stay on your infrastructure can't fall back to an external API. See the hybrid fleet.
- Keep prompts working on every model in the chain. A prompt tuned for one model can break on another, so test them.
- Validate outputs at every level, not only on the primary.
- Log which level answered every request, so you can see the fallback rate.
- Use a circuit breaker to skip a level that keeps failing, instead of waiting for its timeout every time.
The code for retries, circuit breakers and provider fallback is on the reliability patterns page. Gateways like LiteLLM can configure fallbacks across providers without code.
Writing the chain down
For your top three LLM calls, write:
- The chain, level by level.
- What triggers each level.
- What the user sees.
- The eval pass rate of each fallback model.
Run your eval set on each fallback model, and accept a model into the chain only if its pass rate is good enough for that call.
Measuring it
- Fallback rate per level.
- Pass rate per fallback model, from offline evals and sampled production traffic.
- Time spent at each level during incidents.
From the floor
In Agent Forge's production version, the model call iterated over a list of fallback models from the settings: if the call to one model failed, the code moved to the next, and every response had to match a JSON schema before it was accepted.
On day 1 of Build30, the first level of the chain was already there in the client configuration: a timeout and retry options with 3 attempts.