Vector databases for RAG: what they do and how to pick one
A vector database stores embeddings with their text and metadata, and finds the vectors closest to a query quickly, even among millions, using approximate nearest neighbour indexes. For RAG, what matters beyond speed is metadata filtering (for permissions and scope), hybrid keyword and vector search, easy updates and deletes, and fitting into the infrastructure you already run. Start with Chroma or pgvector for most projects, move to a dedicated system like Qdrant, Weaviate, Milvus or a managed service like Pinecone when your scale, filtering or operations needs require it.
Why not just a normal database?
A normal database is built to find exact matches: the row where order_id = 123, or the documents that contain the word "refund".
That is keyword search, and it is fast and precise when the user's words match the document's words.
A vector database is built to find the closest matches in meaning. It stores the vector for each chunk, and for a query vector, it finds the vectors with the highest cosine similarity, the semantic search that understands context and intent.
You could store vectors in any database and compare the query with every vector one by one. For a few thousand chunks, that is fine, and the index internals page shows how to build exactly that in plain Python. For millions of chunks with many queries per second, comparing with every vector is too slow, and that is what vector databases solve.
What a vector database does
- Stores vectors, the original text and metadata together.
- Builds an approximate nearest neighbour (ANN) index, like HNSW, so search time stays low as the collection grows.
- Filters by metadata during search, for example only chunks from documents this user can see, or only the 2026 policies.
- Supports hybrid search, combining vector similarity with keyword search, in many products.
- Handles inserts, updates and deletes without rebuilding everything.
- Scales across machines, replicates for reliability and backs up data, in the server and managed versions.
Main options
Chroma
Open source and easy to start with, it runs inside your Python process or as a server. It is what we use first in the cohort, because it gets a working RAG pipeline running with a few lines, and it is a good fit for lightweight to medium workloads.
pgvector
An extension that adds vector columns and indexes to PostgreSQL. If your application already uses Postgres, this keeps vectors next to the rest of your data, with SQL, joins, transactions, backups and access control you already have. It is often the most practical choice for production systems of moderate size.
Qdrant, Weaviate and Milvus
Open-source databases built specifically for vectors, which you can self-host or use as managed services. They offer strong filtering, hybrid search, quantization and horizontal scaling, and are good choices when vectors are a core part of your system at large scale.
Pinecone
A fully managed, proprietary vector database. You don't run any infrastructure, and you pay based on usage beyond the free tier.
FAISS
A library from Meta for fast similarity search, not a database. It is very fast and flexible for building indexes in your own code, and you handle storage, metadata and updates yourself.
Search engines with vectors
Elasticsearch and OpenSearch, which many companies already run for keyword search, support vector search too, which makes hybrid search natural.
Embedded and local options
LanceDB, Chroma in embedded mode, SQLite with a vector extension, and FAISS run entirely on a laptop or desktop, which is what you need for offline and on-device applications.
How to choose
| Situation | Good starting point |
|---|---|
| Learning, prototypes, small projects | Chroma |
| Your app already runs on PostgreSQL | pgvector |
| Large scale, heavy filtering, vectors at the core | Qdrant, Weaviate or Milvus |
| No infrastructure team, want fully managed | Pinecone or a managed service of the above |
| Already running Elasticsearch or OpenSearch | Their vector search |
| Offline, desktop or air-gapped | Chroma embedded, LanceDB or FAISS |
Questions that decide it:
- How many chunks now, and in two years?
- How many queries per second, and what latency?
- Do you need filters on every query, like permissions?
- Do you need hybrid search?
- Must data stay on your servers?
- What does your team already know how to run?
The pgvector vs Qdrant vs Pinecone comparison goes into more detail.
Things that matter in production
- Metadata filtering performance: filtering by user permissions on every query must stay fast, and some indexes slow down sharply with restrictive filters.
- Updates and deletes: when a document changes, its old chunks must be removed. Keep a document ID on every chunk so you can delete by document.
- Index settings: ANN indexes trade a little accuracy for speed, and their settings decide how much. Defaults are a starting point, measure recall on your data.
- Memory: indexes like HNSW are usually kept in RAM, so vector count times dimensions times bytes per value decides your memory needs. Quantization can reduce it.
- Backups and rebuilds: keep the original text and metadata somewhere you can rebuild the index from, for when you change embedding models.
- Multi-tenancy: if you serve several customers, decide whether each gets its own collection or shares one with a tenant filter.
You may not need a separate vector database
For a few thousand chunks, a NumPy array and a brute-force search is fast enough and has nothing to operate. For applications on Postgres, pgvector avoids a new system. Add a dedicated vector database when you have a concrete reason: scale, filtering performance, or features you need.
From the floor
In the cohort, I chose Chroma for our first RAG builds because it is open source and good for lightweight and medium workloads, and I pointed builders to Pinecone as the proprietary alternative, which charges after a free limit, and to FAISS from Meta as an open-source library for similarity search. Starting with an open tool keeps the whole pipeline on your machine while you learn.
Related
- pgvector vs Qdrant vs Pinecone
- Vector index internals
- Metadata filtering and access control
- Hybrid search