Teardown: what happens when you send a message to a chat assistant
A message to a large chat assistant likely passes through much more than the model: an API gateway with authentication and rate limits, context assembly with history and memory, safety checks on the input, possibly a router choosing a model or reasoning mode, tools such as search or code execution, the model itself on GPU clusters, checks on the output, and logging for quality and abuse monitoring. This is a reasoned design, not a description of any company's internal system.
Likely path
| Step | What likely happens |
|---|---|
| 1 Client and gateway | Authentication, rate limits, account tier |
| 2 Context assembly | Conversation history (resent each turn), saved memory, system instructions, uploaded files |
| 3 Input safety | Classifiers for abuse and policy |
| 4 Routing | Which model or mode handles this message, fast or reasoning, from settings and the request |
| 5 Tools | Web search, code execution, file reading, connected apps, each returning results into the context |
| 6 Inference | Prefill and decode on GPU clusters with batching and KV caching |
| 7 Output | Checks, then streaming tokens to the client |
| 8 Logging | Evaluation, abuse detection, preference feedback |
Inside inference
The prompt is processed in one parallel pass (prefill), filling the KV cache. Then tokens are generated one at a time (decode), each one reading the cache, with many users' requests batched together on the same GPUs. Prefill decides time to first token, and decode speed decides how fast the answer streams.
What you can observe yourself
- Long conversations get slower and more expensive to run, because history is resent and the cache grows.
- Asking the same question twice gives different wording (nondeterminism).
- Some answers show a search step or a thinking step, which are tool and reasoning modes.
- Being asked to choose between two answers is preference data collection.
- Very long chats start to lose track of early details, which is why starting a fresh chat often helps.
What this means for your own product
Your assistant needs the same zones at a smaller scale: a gateway, context assembly, input checks, the model, output checks and tracing. See the LLM inference pipeline for each zone in detail.
From the floor
In the inference pipeline lecture, I traced the question "what is the delivery status of my order, and what's your policy if it arrives damaged?" from request to response: it has two parts, one that needs the order database and one that needs the policy documents, and the system has to plan both before the model writes a single answer.
I've also written this up as a blog called "A day in the life of a token".