RAG architecture end to end: how retrieval-augmented generation works in production
Retrieval-augmented generation (RAG) gives a model information it was never trained on by finding the relevant pieces of your documents at question time and putting them into the prompt. A production RAG system has two pipelines: an ingestion pipeline that parses, chunks, embeds and indexes your documents with metadata, and a query pipeline that rewrites the question if needed, retrieves candidates with hybrid search, filters by permission, reranks, and asks the model to answer only from the retrieved text with citations. Most RAG failures are retrieval failures, so you evaluate retrieval and generation separately.
Why RAG exists
An LLM knows what it was trained on, up to its knowledge cutoff. It doesn't know today's weather, your company's refund policy, your insurance products, or the contract you signed last week. Ask it anyway, and it either says it doesn't know or produces something that sounds right.
You have two ways to add that knowledge. You could fine-tune the model on your data, but fine-tuning is expensive, and the moment you finish, the model is out of date: if you fine-tune on data up to the 16th, it knows nothing about what changes on the 17th. Or you could leave the model as it is, keep your data in a place you can search, and give the model the relevant pieces when a question comes in.
That second approach is RAG. Your data stays outside the model, you can update it any time, and the model uses what it's good at, understanding language and writing answers, on information you control.
A library analogy
Think about writing an essay on a topic you know nothing about. You know how to write an essay: how to structure it, summarise, conclude. You don't know the topic. So you go to the library, find the right books, read the relevant parts, and then write the essay in your own words.
Finding and reading the books is retrieval. Writing the essay is generation. The LLM already knows how to write, and RAG is the trip to the library.
RAG and the model work together
RAG doesn't change the model. The weights stay exactly the same, and no training happens. The retrieved text is simply added to the prompt, and the model reads it the same way it reads anything else you send.
That means each part can be improved separately: better retrieval gives the model better material, and a better model makes better use of the same material. It also means neither works well alone. Retrieval without a model gives you search results, and a model without retrieval gives you answers from memory.
Ingestion pipeline
This runs when documents are added or updated.
- Load: collect documents from their sources, like file shares, SharePoint, Confluence, databases, websites or uploads.
- Parse: turn PDFs, Word files, HTML, slides and scans into clean text, keeping structure like headings and tables.
- Chunk: split the text into pieces small enough to retrieve precisely and large enough to make sense on their own.
- Enrich: attach metadata to each chunk, like the source document, page, section, date, department and who is allowed to see it.
- Embed: turn each chunk into a vector with an embedding model.
- Index: store vectors, text and metadata in a vector database, and the text in a keyword index for hybrid search.
It also needs to handle updates and deletions, so a changed policy replaces the old chunks instead of sitting next to them.
Query pipeline
This runs for every question.
- Understand the question: optionally rewrite it, split it into sub-questions, or add context from the conversation.
- Retrieve: find candidate chunks with vector search, keyword search or both.
- Filter: keep only chunks this user is allowed to see, and that match filters like product or date.
- Rerank: reorder the candidates by how well they answer the question, and keep the best few.
- Build the prompt: instructions, the selected chunks with their source IDs, and the question.
- Generate: the model answers from the chunks, cites them, and says when they don't contain the answer.
- Check: validate citations and format before returning the answer.
A minimal RAG in code
The smallest useful version fits in a few dozen lines with LangChain, a local embedding model and Chroma, the stack we use in the cohort to start.
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
docs = PyPDFLoader("data/annual_report.pdf").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200).split_documents(docs)
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
store = Chroma.from_documents(chunks, embeddings, persist_directory="index")
question = "What are the main risks in the company's international supply chain?"
results = store.similarity_search(question, k=5)
context = "\n\n".join(f"[{i}] (page {d.metadata.get('page')}) {d.page_content}" for i, d in enumerate(results))
prompt = f"Answer only from the sources below and cite them like [0]. If they don't contain the answer, say so.\n\n{context}\n\nQuestion: {question}"
Send prompt to any model, as on the API page.
This version works for a demo, and every page in this section is about what it takes to make it work on real documents, real questions and real users.
Naive RAG and production RAG
| Naive RAG | Production RAG |
|---|---|
| Text extracted with a basic PDF loader | Layout-aware parsing, tables and scans handled |
| Fixed-size chunks | Chunks that follow document structure, with context |
| Vector search only | Hybrid search plus reranking |
| No metadata | Metadata for filtering, permissions, freshness and citations |
| Top 5 chunks into the prompt | Fewer, better chunks, ordered and labelled |
| Answers without sources | Verified citations and "I don't know" when needed |
| Tested on a few questions | Evaluated on a test set, retrieval and answers separately |
| Built once | Updated as documents change, monitored in production |
Where RAG systems fail
Most failures happen before the model writes a word: the document was parsed badly, the answer was split across two chunks, the right chunk wasn't retrieved, or it was retrieved but ranked too low to be included. The rest happen in generation: the model ignored the context, mixed two sources, or answered when it should have said it didn't know. The failure modes page lists them with fixes, and the evaluation page shows how to measure each half.
How this section is organised
The pages follow the order of the pipelines: parsing, chunking, embeddings, vector databases and their internals, hybrid search, reranking, query rewriting, metadata and access control, then GraphRAG and agentic RAG for harder cases, and finally evaluation, failure modes and RAG for regulated data.
From the floor
In the RAG fundamentals session, a builder from the insurance domain asked the question many companies ask: their data isn't public, so can an LLM work with it at all? That is exactly the case RAG is for: the company keeps its own data, the model never needs to be trained on it, and new policies are added to the index the day they're written.
The first production RAG project in our cohort answered questions about companies' annual financial reports. We built it locally first with LangChain, a sentence-transformers embedding model, Chroma and a local Llama model through Ollama, then followed the same production phases as every project: containerization, CI/CD with GitHub Actions, orchestration, monitoring with Prometheus and Grafana, and infrastructure on AWS.