Exact values pattern: stop the LLM from rewriting your numbers
When an answer contains values that must be exact (sales totals, prices, order IDs, dates, balances), get them with code (a SQL query, an API call, a calculation) and give them to the model as fixed values to place in the answer. Then check that every value in the final answer matches a value from the source, and for dashboards and tables, render critical values directly from the data and let the model write only the words around them. Retrieval doesn't make generation exact: exact values belong to code, and the model's job is the language.
What goes wrong
A company wants its people to ask questions about their own data through a simple chatbot. The CEO asks for one salesperson's sales over the last six months. The CSV says $44,000, retrieval finds the right rows, and the model writes "$40,000" in a fluent, confident sentence.
Nothing crashed. The answer looks right. And it's wrong in the one place that matters.
This happens because a language model generates numbers as tokens, the same way it generates words. It can copy a value correctly most of the time and still change a digit, round, or add two numbers wrongly, especially when there are many numbers in the context.
How the pattern works
- Detect that the question needs structured data, through the router or the agent's tool choice.
- Get the values with code. For text-to-SQL, the model writes the query and code executes it, with validation and read-only credentials.
- Pass the result to the model as structured data, with an instruction to use the values exactly as given.
- After generation, extract the numbers and IDs from the answer and compare them with the source values. Block or repair mismatches.
- For tables and dashboards, render values from data directly, and let the model write only the commentary.
import re
def numbers_in(text: str) -> set[str]:
return {n.replace(",", "") for n in re.findall(r"\d[\d,]*(?:\.\d+)?", text)}
def check_exact(answer: str, source_values: dict[str, float | int | str]) -> list[str]:
allowed = {str(v).replace(",", "") for v in source_values.values()}
return sorted(numbers_in(answer) - allowed)
source = {"sales_6m_usd": 44000, "deals": 17, "months": 6}
for answer in [
"Ravi closed 17 deals worth $44,000 in the last 6 months.",
"Ravi closed 17 deals worth $40,000 in the last 6 months.",
]:
bad = check_exact(answer, source)
print("PASS" if not bad else f"FAIL {bad}", "|", answer)
A real check also handles formatting (44k, 44,000.00, dates), and decides what to do on failure: regenerate with a stricter instruction, fall back to a template answer, or show the table without commentary.
Where knowledge graphs fit
A knowledge graph can help find the right entities and relationships, for example which salesperson, which region, which product line. The final number still comes from the source query, and the check above still applies. See GraphRAG.
Templates for the most critical answers
For answers where any mistake is unacceptable, like account balances or invoice totals, skip generation for the value entirely: build the sentence from a template with the value inserted by code, and use the model only for the parts that need language.
Measuring it
Track the mismatch rate between values in answers and values from tools, per route. After the check is in place, the mismatch rate reaching users should be zero, and the rate caught by the check tells you how often the model tries to change numbers.
From the floor
I used this exact example in the AI system design session: enterprises want to query their existing data through a simple chatbot, and in the example, the CSV said $44,000 for six months of one person's sales, and the chatbot, combining that data with generation, could produce a different number in a conversational answer, which is one of the most common complaints about these systems.
At Dextar, the knowledge engine we're building for company data uses knowledge graphs alongside retrieval, so that exact numbers, for example from sales data, are picked up from the data itself and the LLM only turns them into a conversational answer.