Reranking in RAG: why and how to add a cross-encoder
Retrieval with embeddings is fast because the question and each chunk are embedded separately, and that same separation makes it imprecise: it finds chunks about the same topic, not necessarily the chunks that answer the question. A reranker reads the question and each candidate chunk together and scores how well the chunk answers it. The standard pattern is to retrieve 20 to 50 candidates with hybrid search, rerank them, and send only the top 3 to 5 to the LLM, which usually improves answers and reduces tokens at the same time.
What problem reranking solves
When a user asks a question, a basic RAG system pulls whatever is close in the vector space, without really checking what the user wants. Ask about Tesla's autonomous driving and you may get chunks about Apple's car project too, because both are about cars and technology. Ask "what is the refund period for electronics?" and you may get "refund periods vary by category", which is about the topic and doesn't contain the answer.
The user doesn't want everything related, they want the relevant few, in the right order. Search engines solved this long ago: when you search YouTube for Python videos, there are thousands of matches, and a ranking algorithm decides which ones appear first. Reranking brings that idea to RAG.
Bi-encoders and cross-encoders
The embedding models used for retrieval are bi-encoders. They encode the question and the chunk separately into vectors, and compare the vectors. Because chunk vectors are computed once in advance, search over millions of chunks takes milliseconds. But the model never sees the question and the chunk together, so it can only judge general similarity.
A cross-encoder takes the question and one chunk together as a single input, and the model's attention can look at every word of the question against every word of the chunk. It outputs a relevance score. That is much more accurate, and much slower, because it has to run once for every candidate chunk at query time.
So you use both: the bi-encoder (and BM25) to find candidates quickly, and the cross-encoder to judge them carefully.
Retrieve-then-rerank pattern
- Retrieve a generous set of candidates, for example the top 30 to 50 from hybrid search.
- Score each candidate against the question with the reranker.
- Sort by the reranker's score.
- Keep the top 3 to 5, optionally only those above a score threshold.
- Send those to the LLM.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
question = "What is the refund period for electronics?"
candidates = [
"Refund periods vary by product category.",
"Electronics can be returned for a full refund within 10 days of delivery.",
"Our electronics range includes laptops, phones and accessories.",
]
scores = reranker.predict([(question, c) for c in candidates])
for score, chunk in sorted(zip(scores, candidates), reverse=True):
print(round(float(score), 2), chunk)
The chunk that contains the actual answer should come first, even though all three are about refunds or electronics.
Kinds of rerankers
- Open cross-encoders: models like the ms-marco cross-encoders or the BGE reranker family, which you run yourself. Small ones run on a CPU, larger ones need a GPU for low latency.
- Reranking APIs: hosted services like Cohere Rerank, Voyage and Jina, where you send the question and candidates and get scores back.
- LLM rerankers: ask an LLM to score or order the candidates. Flexible, and you can give it instructions like "prefer the most recent policy", at a higher cost and latency.
- Late-interaction models: ColBERT-style models keep a vector per token and compare at a finer level than bi-encoders, a middle ground between speed and precision.
Cost and latency
Reranking adds a step to every query. Reranking 30 candidates with a small cross-encoder typically adds tens to a few hundred milliseconds, depending on the model and hardware, and an API adds network time and a per-query cost.
In return, you send fewer, better chunks to the LLM, which reduces input tokens and often improves the answer more than switching to a bigger LLM would. Measure both sides on your system before deciding.
Beyond relevance: business rules
Ranking can include more than relevance:
- Freshness: prefer the newest version of a policy.
- Authority: prefer official documents over forum posts.
- Diversity: avoid five near-identical chunks from the same document.
- Metadata: boost chunks from the product or region the user is asking about.
Some of this belongs in metadata filters before retrieval, and some in a final scoring step after reranking.
How to tell if reranking helps
On your evaluation set, compare retrieval without and with reranking:
- Precision at 3 or 5: of the chunks you send to the LLM, how many are relevant?
- MRR: how high does the first relevant chunk rank?
- Answer quality and faithfulness on the full pipeline.
- Latency and cost per query.
If retrieval already returns the right chunk in first place almost every time, reranking adds little. If the right chunk is often in the top 30 but not the top 5, reranking is the fix.
From the floor
In our advanced RAG session, the dataset had short documents about Tesla's reports, autonomous driving and Apple's financials. A basic pipeline asked about Tesla pulled the Apple chunks too, and a question about self-driving cars brought both companies' content, without looking at what the user wanted. The session was about adding reranking and structured retrieval with metadata to fix exactly that, so the pipeline ranks results by relevance instead of returning everything that is merely related.