Context compression and summarisation
Context compression means sending the model less text while keeping the information it needs. For conversations, keep recent turns in full and replace older ones with a running summary. For documents, retrieve and rerank so only relevant parts are sent, and extract the facts you need from long sources. For agents, clear old tool results and compact the run into a short state when it grows. Every compression step can drop something important, so test compressed context against full context on your evaluation set.
Why compress
Long context costs money on every call, slows the first token, and makes the model less accurate when the important details are buried among irrelevant ones. In a long chat or a long agent run, the context keeps growing until it hits the window limit, and something has to give.
Compression is how you decide what gives, instead of letting the application cut off the oldest messages blindly.
What to compress
There are three common sources of bloat:
- Conversation history, which grows every turn.
- Documents and retrieved chunks, which are often longer than the part that matters.
- Tool results, which can return whole web pages, files or database tables.
Conversation history
Sliding window
Keep only the last N turns. It is simple and cheap, and it loses anything said before the window, including the user's original goal.
Rolling summary
Keep the last few turns in full, and maintain a summary of everything before them. When the conversation grows past a limit, summarise the oldest turns into the summary.
Tell the summariser exactly what to keep: the user's goal, decisions made, facts the user gave (names, numbers, preferences), open questions and commitments the assistant made. Generic summaries tend to keep the topic and drop the specifics, and the specifics are what the next turn needs.
Update the conversation summary below with the new messages.
Keep: the customer's goal, order IDs, amounts, dates, decisions made, promises made by the assistant, and open questions.
Drop: greetings, repeated questions, small talk.
Write at most 150 words.
<summary>{current_summary}</summary>
<new_messages>{messages}</new_messages>
Structured state
For task-oriented conversations, keep a small structured record instead of, or alongside, a summary.
{"goal": "replace damaged laptop charger", "order_id": "ORD-123456", "status": "awaiting photo of damage", "promised": "reply within 24 hours"}
Structured state is easier to check and harder to distort than prose.
Documents
- Retrieve and rerank: send the 3 to 5 most relevant chunks, not the top 20.
- Trim chunks: remove headers, footers, navigation text and boilerplate before indexing.
- Extract: for long documents, a first pass can extract only the facts relevant to the question, and the second pass answers from those facts.
- Map and reduce: for summaries of very long documents, summarise each section, then summarise the summaries.
Tool results
Tools often return far more than the model needs.
- Return only the needed fields from APIs and databases, and limit the number of rows.
- Convert HTML to clean text, and drop navigation and ads.
- Truncate long outputs with a note, like "showing 10 of 340 results", so the model knows more exist.
- After the model has used a tool result, replace it in the history with a short note of what it found.
Agent compaction
Long agent runs need compaction: when the context gets close to a limit, the agent summarises the run so far, the goal, what has been done, what was learned, what's left, and continues from that summary with a fresh context. Coding agents like Claude Code do this, and also keep plans and decisions in files, so they don't depend on remembering them.
Two tips make compaction safer:
- Keep the original task and hard constraints verbatim, never summarised.
- Store important details, like file paths, IDs and decisions, in external notes the agent can read again.
Prompt compression methods
There are also methods that compress the prompt text itself, by removing tokens a small model predicts are less important. They can cut tokens significantly, but they make prompts harder to read and debug, and the quality effect depends on the task, so test them carefully before using them.
Test what you lose
Compression is a trade-off between cost and information. For each strategy, run your evaluation set with full context and with compressed context, and compare accuracy, cost and latency. Pay special attention to questions that depend on details from early in a conversation or document, because those are what compression loses first.
From the floor
I gave Claude Code the errors in my project and asked it to fix them and explain what went wrong, and it did, but as we keep working on the project and the context fills up, it can miss some of the earlier instructions. So I give it smaller tasks, one component at a time, and check what it changed before moving forward, which is context management done by the person instead of the tool.