How to choose an embedding model for RAG
The embedding model decides which chunks your RAG system considers similar to a question, so it has a direct effect on retrieval quality. Use the MTEB leaderboard to shortlist 3 or 4 models that fit your languages, budget and hosting needs, then test them on 50 to 100 of your own questions with known answer passages, and pick the one with the best recall at an acceptable cost. Remember that switching models later means re-embedding everything, so record which model and version built each index.
Why the embedding model matters
In a RAG system, the question and every chunk are turned into vectors by the same embedding model, and retrieval returns the chunks whose vectors are closest to the question's. If the model doesn't place "my parcel never arrived" near "missing shipments", the right chunk isn't found, and no LLM can answer from a chunk it never received.
Different models are trained on different data, for different languages and tasks, so they disagree about what is similar, especially for specialised vocabulary like legal, medical or financial terms.
What to compare
Retrieval quality on your data
This is the only measure that really counts. General benchmarks tell you how a model does on many public datasets, not on your documents and your users' phrasing.
Languages
If your users write in Hindi, Tamil or a mix of English and Hindi, check that the model was trained for those languages, and test with real queries in them. Multilingual models can also retrieve across languages, matching a Hindi question to an English document, which may be exactly what you need, or something to watch for.
Maximum input length
Every embedding model has an input limit in tokens. Text beyond it is usually cut off without an error, so a 1000 token chunk embedded by a 512 token model only represents its first half. Make sure your chunks fit.
Dimensions
Embeddings range from a few hundred to a few thousand dimensions. More dimensions can capture more nuance, and they cost more storage, memory and search time. Some models support shortening their vectors (Matryoshka representation learning), so you can store 512 dimensions instead of 3072 with a small loss of quality.
Cost and speed
Hosted embedding APIs charge per token, and you pay to embed your whole corpus once, every update, and every query. Embedding is usually cheap compared with generation, but re-embedding a large corpus when you change models adds up. Open models are free to use, and you pay for the hardware to run them, which for small models can be a CPU.
Hosting and privacy
If documents can't leave your infrastructure, the embedding model has to run on it too, because embedding sends the full text of every chunk to the model. Many strong open models, like the BGE, E5, Nomic, GTE and Qwen3-Embedding families, can run locally.
Domain
For specialised domains, look for models trained or fine-tuned on similar text, and if none fits well, consider fine-tuning an open embedding model on pairs of questions and passages from your own data.
Types of embedding models
- Hosted APIs: OpenAI, Google, Cohere, Voyage and others offer embedding models through an API. Easy to start, strong quality, data leaves your servers.
- Open models: available on Hugging Face, run with sentence-transformers or an inference server. Full control, you run the infrastructure.
- Small, fast models: like all-MiniLM-L6-v2 (384 dimensions), useful for prototypes, learning and CPU-only environments, weaker than larger models on hard retrieval.
- Multilingual models: trained on many languages, needed if your users or documents aren't only in English.
- Sparse and multi-vector models: models like SPLADE produce sparse, keyword-like vectors, and late-interaction models like ColBERT keep one vector per token for finer matching. They can improve retrieval, and they need support in your vector database.
Using the MTEB leaderboard
The Massive Text Embedding Benchmark (MTEB) leaderboard on Hugging Face compares embedding models across many tasks and languages. For RAG, look at the retrieval scores, not the overall average, and filter by the languages you need, model size and maximum input length.
Use it to build a shortlist. Rankings change often, several models are close to each other, and some models may have seen benchmark data during training, so the top of the leaderboard is not automatically the best for you.
Test on your own data
- Collect 50 to 100 real questions.
- For each, mark the chunk or chunks that contain the answer.
- Embed your corpus with each candidate model, using the same chunking.
- For each question, retrieve the top 5 and top 10 chunks.
- Measure recall at 5 and 10 (how often the right chunk appears) and MRR (how high it ranks).
- Compare cost, speed and storage, and pick the best trade-off.
def recall_at_k(results: list[list[str]], relevant: list[set[str]], k: int) -> float:
hits = sum(1 for got, want in zip(results, relevant) if want & set(got[:k]))
return hits / len(relevant)
def mrr(results: list[list[str]], relevant: list[set[str]]) -> float:
total = 0.0
for got, want in zip(results, relevant):
for rank, chunk_id in enumerate(got, start=1):
if chunk_id in want:
total += 1 / rank
break
return total / len(relevant)
results = [["c3", "c7", "c1"], ["c2", "c9", "c4"], ["c8", "c5", "c6"]]
relevant = [{"c1"}, {"c2"}, {"c4"}]
print(recall_at_k(results, relevant, k=3), round(mrr(results, relevant), 3))
Practical rules
- Use the same model for documents and queries. Some models also expect a prefix, like "query: " and "passage: ", so read the model card.
- Normalise vectors if the model expects cosine similarity, and use the distance metric the model was trained with.
- Record the model name and version with every index, so you know what built it.
- Plan for re-embedding. A better model will come out, and switching means embedding every chunk again and rebuilding the index, so keep the original text and a script that rebuilds the index from scratch.
From the floor
In the cohort RAG builds, we start with sentence-transformers' all-MiniLM-L6-v2, a small model that runs on a laptop, because it lets builders build and debug the whole pipeline locally without an API key. It is the right model for learning how retrieval works, and the first thing to test against stronger models when the project moves to real documents.