Query rewriting and decomposition for RAG
Users don't write questions in the words your documents use, and in a conversation they often don't write complete questions at all ("what about for laptops?"). Query rewriting uses a fast LLM call to turn the user's message into one or more better search queries before retrieval: a standalone question from the conversation, extra terms and synonyms, a hypothetical answer to search with, or several sub-questions for a complex request. It adds a little latency and cost per query, so use it where your evaluation shows retrieval failing on the raw question.
Why the raw question often retrieves badly
- Follow-up questions lack context. After asking about the refund policy for phones, a user asks "and for laptops?". Searching "and for laptops?" retrieves nothing useful.
- Vocabulary differs. Users say "money back", documents say "refund". Users say "my screen is broken", documents say "display damage".
- Questions are vague. "Tell me about benefits" could mean insurance, leave or pensions.
- Questions are compound. "Compare the leave policy for interns and full-time employees in India and the UK" contains four separate lookups.
- Questions are short. Two or three words give embeddings little to work with.
Techniques
Make the question standalone
In a conversation, rewrite the latest message into a complete question using the history.
Rewrite the user's last message as a standalone search query, using the conversation for context.
Return only the query.
Conversation:
User: What's the refund period for phones?
Assistant: Phones can be returned within 7 days of delivery.
User: and for laptops?
The output, "What is the refund period for laptops?", retrieves the right chunk. This is the single most useful rewriting step for chat-based RAG.
Expand the query
Add synonyms, related terms and expanded abbreviations, or generate several paraphrases of the question and search with all of them, merging the results. Multi-query retrieval like this improves recall when users and documents use different words, at the cost of several searches.
Hypothetical document embeddings (HyDE)
Ask the LLM to write a short hypothetical answer to the question, without retrieval, and search with the embedding of that answer instead of the question. Answers look more like documents than questions do, so they often land closer to the right chunks in embedding space. The hypothetical answer is only used for searching, never shown to the user, because it may be wrong.
Decompose complex questions
Split a compound question into sub-questions, retrieve for each, and answer from the combined results.
"Compare the leave policy for interns and full-time employees in India and the UK" becomes four lookups, one per combination, followed by a comparison. For questions where later steps depend on earlier answers ("Who manages the team that owns the payments service, and what is their escalation process?"), the sub-questions run in sequence, which is where retrieval starts to look like an agent. See agentic RAG.
Step-back questions
For very specific questions, also search with a more general version ("What are the rules for carrying forward leave?" alongside "Can I carry forward 7 days of leave from 2025?"), so the general rule is retrieved along with any specific mention.
Extract filters
Pull structured filters out of the question and apply them as metadata filters: "the 2025 travel policy for the UK office" becomes a search for "travel policy" filtered to year 2025 and region UK. Use structured output for this so the filters are valid.
Route the question
Decide where a question should go before searching: the HR documents, the product manuals, a database query, a web search, or no retrieval at all for greetings and general questions. A small model or simple rules can do this cheaply.
Costs and risks
Each rewriting step is an extra model call, adding latency and cost to every query. Use a small, fast model for rewriting, keep the prompts short, and skip rewriting when the question is already clear.
Rewriting can also go wrong: the model can change the meaning of the question, add assumptions, or drop an important detail like a product code. Always keep the original question for the final answer step, and consider searching with both the original and the rewritten query.
How to decide what to use
Look at your retrieval failures on the evaluation set:
| Failure | Technique |
|---|---|
| Follow-up questions retrieve nothing | Standalone rewriting |
| Users and documents use different words | Expansion, multi-query or HyDE |
| Compound questions get partial answers | Decomposition |
| Results from the wrong year, product or region | Filter extraction |
| Questions need different sources | Routing |
Add one technique at a time and measure the change in recall and answer quality.
From the floor
In the customer support agent, routing came before retrieval: based on the query, the router decides whether to answer from the cache, send it to an external data source or RAG integration, or send it to the LLM, so similar queries don't all end up as new LLM calls. Query rewriting sits in the same place in the flow, between the user's raw question and the retriever.