Data quality for retrieval: what to fix before tuning RAG
Before tuning retrieval, clean the corpus: remove duplicates and near duplicates, keep only current versions (or mark versions in metadata), fix parsing errors, fill missing metadata (dates, owners, document types), and find contradictions between documents. Retrieval can only be as current and consistent as the documents it searches. A day spent cleaning the corpus often improves answers more than a week spent changing embedding models.
A common failure
Three versions of the same refund policy sit in the index. Retrieval returns the oldest one, because its wording happens to match the customer's question best. The model answers faithfully from it, and the answer is wrong. No reranker or prompt fixes that, because the problem is in the data.
Issues and how to find them
| Issue | What it does | How to detect it |
|---|---|---|
| Duplicates | Crowd out other results, waste context | Content hashes, embedding similarity above a high threshold |
| Stale versions | Old answers retrieved | Version and date metadata, superseded flags |
| Parsing errors | Tables and columns turned into nonsense | Sample review, text quality heuristics (odd character ratios, very short lines) |
| Missing metadata | Filters can't work | Required field checks at ingestion |
| Contradictions | Different answers depending on which chunk wins | Compare passages on the same topic with an LLM, reviewed by a person |
| Boilerplate | Headers, footers, disclaimers retrieved instead of content | Repeated text across many documents |
A duplicate and staleness report
import hashlib
from collections import defaultdict
docs = [
{"id": "refund-v1", "title": "Refund policy", "version": 1, "text": "Refunds within 30 days."},
{"id": "refund-v3", "title": "Refund policy", "version": 3, "text": "Refunds within 14 days in the EU."},
{"id": "refund-v2", "title": "Refund policy", "version": 2, "text": "Refunds within 21 days."},
{"id": "faq-copy", "title": "FAQ", "version": 1, "text": "Refunds within 30 days."},
{"id": "shipping", "title": "Shipping", "version": None, "text": "Ships in 2 days."},
]
def norm_hash(text: str) -> str:
return hashlib.md5(" ".join(text.lower().split()).encode()).hexdigest()
by_hash = defaultdict(list)
for d in docs:
by_hash[norm_hash(d["text"])].append(d["id"])
duplicates = [ids for ids in by_hash.values() if len(ids) > 1]
by_title = defaultdict(list)
for d in docs:
by_title[d["title"]].append(d)
stale = [d["id"] for group in by_title.values() if len(group) > 1
for d in group if d["version"] != max(x["version"] or 0 for x in group)]
missing = [d["id"] for d in docs if d["version"] is None]
print("duplicates:", duplicates)
print("stale versions:", stale)
print("missing version metadata:", missing)
Real corpora need near-duplicate detection too (embedding similarity or MinHash), because copies are rarely exact.
Fixing them
- Duplicates: keep one canonical copy, record the others as aliases.
- Stale versions: remove superseded versions from the index, or keep them with a flag and filter them out by default.
- Parsing errors: fix the parser for that document type, then re-ingest from the raw files.
- Missing metadata: make the fields required at ingestion, and backfill.
- Contradictions: send them to the document owners, because only they know which one is right.
Measuring it
- Share of the corpus with complete metadata.
- Number of known duplicates, stale versions and contradictions, trending down.
- Retrieval recall before and after a cleanup, on the same eval set.
From the floor
In the Build30 RAG session, the evaluation step removed stop words like "the" and "with" before checking whether the answer's words came from the retrieved chunks, because words that add no meaning would otherwise make any answer look grounded, a small example of cleaning data before measuring with it.