AI evals and reliability: 20 plays to catch LLM failures before your users do
An eval is a test for an AI feature: you give the system an input, look at what it produced, and decide if it passed or failed. Normal software tests check one exact answer, but AI outputs change from run to run, so an eval checks a rule instead, like "did the bot quote the right refund window?" or "did the agent update the order before saying it was done?". A team with evals can change a prompt, swap a model or tune retrieval and know within minutes whether things got better or worse, and a team without evals finds out from its users.
Why evals decide whether AI ships
Most AI demos work. Most AI products fail somewhere between the demo and production, and the reason is rarely the model. The team can't tell whether a change made things better or worse, so every prompt edit is a guess, every model upgrade is a risk, and every complaint starts a debate instead of a fix.
Evals turn those guesses into numbers you can act on. "The handoff check went from 74% to 91% after the prompt change" is a decision. "It feels better" is not.
What an eval is made of
Every eval has the same parts:
- An input: a real user question, a document, a task for an agent.
- The system's output, and ideally the full trace of how it got there.
- A check: code (does the JSON parse, is the date correct, did the test pass) or a model acting as a judge, validated against a person.
- A result: pass or fail, collected across many inputs into a pass rate.
The hard part isn't writing the check. It's knowing which checks to write, and that comes from reading real traces and finding where your system fails.
How this playbook is organised
These are the 20 plays from my AI Evals and Reliability Playbook, in four sections, plus four supporting pages. Most of the method comes from Hamel Husain and Shreya Shankar (their AI evals course and open evals FAQ). The RAG framework is Jason Liu's, the agent metrics come from the Sierra τ-bench team and Anthropic's engineering blog, and the coding agent material comes from SWE-bench and OpenAI's audits of it.
Section 1: find the failures
- Why MMLU and GPQA scores don't tell you if your app works
- What a production trace records, span by span
- How to find your failure modes in 30 to 100 traces
- Why one domain expert beats five annotators
- How to pick which traces to read
Section 2: measure them correctly
- Why pass or fail beats a 1 to 5 score
- How to check if your LLM judge is right
- How to stop a judge prompt from overfitting
- Why BERTScore, ROUGE and "helpfulness" mislead you
- How to test if a model knows when to say "I don't know"
Section 3: evaluate specific systems
- How to evaluate RAG in two separate halves
- How to choose chunk size with evals
- How to evaluate a coding agent
- How to evaluate an agent end to end and step by step
- How a transition failure matrix finds the broken step
- How to evaluate multi-turn conversations
Section 4: run it in production
- Guardrails vs evaluators: what runs inline and what runs later
- Why you should build your own annotation tool
- How CI gates and production monitoring work together
- When to retire an eval
Supporting pages
- Eval-driven development
- Building an eval set from production traffic
- Online evaluation and A/B tests
- Reliability patterns: retries, fallbacks, timeouts and circuit breakers
Where to start
If you have nothing yet, do these four things in order:
- Log full traces for every request (play 2).
- Read 30 to 100 of them and write down what went wrong (play 3).
- Write a pass or fail check for the most common failure (play 6).
- Run that check on every change before it ships (play 19).
That loop, repeated, is most of what evals are. The other plays make each step faster and more trustworthy.
Evals and reliability
Evals tell you whether outputs are good. Reliability is whether the system keeps producing them when things go wrong around the model: a provider has a bad day, a tool times out, traffic doubles. The reliability patterns page covers retries, fallbacks, timeouts and circuit breakers, and LLMOps covers running all of it in production.
From the floor
In a Build30 RAG session, a builder asked which evaluation framework and which metrics to use to justify that the pipeline's answers were correct, and mentioned promptfoo and DeepEval. My honest answer was that evaluation needs its own session because there are many parts to it, and that I was working on the eval part myself. This playbook is that work: it starts from finding your own failures before choosing any framework, because a framework can run checks, and only your traces can tell you which checks matter.
When we built the sentiment classification pipeline with a local LLM, the rule I gave was to work on the evaluation methods in parallel with the model, not after it, so you can see while building whether the approach is going right and tweak it.