Semantic caching for LLM apps
A semantic cache stores past questions as embeddings with their answers. When a new question arrives, it looks for a stored question above a similarity threshold, and if it finds one, returns that answer without calling the model (a cache hit), otherwise it calls the model and stores the result (a miss). It saves cost and latency on repetitive traffic, and it must never serve an answer that depends on user-specific or time-sensitive data.
Why cache LLM answers
Thousands of users ask "how do I reset my password?" in slightly different words, and each one triggers a full model call. Model inference is the most expensive step in the pipeline, so answering the same question again from a cache saves money and returns the answer in milliseconds.
Three kinds of caching
| Cache | Matches on | Saves |
|---|---|---|
| Exact-match cache | The identical (normalised) question | The whole call |
| Semantic cache | A question with the same meaning | The whole call |
| Prompt caching (provider side) | An identical prompt prefix | Part of the prefill cost and time |
They work together: exact match first (cheapest), semantic match second, and prompt caching for the calls that still reach the model.
How a semantic cache works
- Normalise the question (lowercase, trim spaces) and check the exact-match cache, a hash lookup in something like Redis.
- On a miss, embed the question.
- Search the cache's vector index for the nearest stored question, within the same scope.
- If the similarity is above your threshold, return the stored answer.
- Otherwise, call the model, return the answer, and store the question, its embedding and the answer if it's cacheable.
import hashlib
import math
import re
def embed(text: str) -> dict[str, float]:
words = re.findall(r"[a-z]+", text.lower())
vec = {w: words.count(w) for w in set(words)}
norm = math.sqrt(sum(v * v for v in vec.values()))
return {w: v / norm for w, v in vec.items()}
def cosine(a: dict, b: dict) -> float:
return sum(a[w] * b.get(w, 0.0) for w in a)
class SemanticCache:
def __init__(self, threshold: float = 0.8) -> None:
self.exact: dict[str, str] = {}
self.entries: list[tuple[str, dict, str]] = []
self.threshold = threshold
def key(self, question: str, scope: str) -> str:
return hashlib.sha256(f"{scope}|{' '.join(question.lower().split())}".encode()).hexdigest()
def get(self, question: str, scope: str) -> str | None:
if (hit := self.exact.get(self.key(question, scope))) is not None:
return hit
q = embed(question)
best = max(
((cosine(q, e), answer) for s, e, answer in self.entries if s == scope),
default=(0.0, None),
)
return best[1] if best[0] >= self.threshold else None
def put(self, question: str, scope: str, answer: str) -> None:
self.exact[self.key(question, scope)] = answer
self.entries.append((scope, embed(question), answer))
cache = SemanticCache()
cache.put("How do I reset my password?", "en|v3", "Go to Settings, then Security, then Reset password.")
for q in ["how do I reset my password?", "How can I reset my password", "How do I delete my account?"]:
print(f"{q!r:<38} -> {cache.get(q, 'en|v3')}")
The toy embedding here counts words, so you can run it without a model. In production, use a real embedding model and a vector store, often Redis with vector search or the same vector database you use for retrieval.
What's safe to cache
Cache hits are only safe for answers that are the same for everyone at that moment.
- Cacheable: "What's your refund policy?", "How do I reset my password?", "What are your opening hours?"
- Not cacheable: "Where's my order?", "What's my balance?", anything that depends on who is asking, live data or the time.
Decide cacheability explicitly, with a rule or a classifier, rather than caching everything and hoping.
Scope the cache key
Include in the scope everything that changes the right answer:
- Tenant or customer, so one company's answer never reaches another.
- Language.
- Product or plan.
- Prompt version and document version, so a policy update doesn't keep serving the old answer.
Setting the threshold
Too low, and different questions get the same answer ("How do I reset my password?" and "How do I delete my account?" are close in embedding space). Too high, and you rarely hit.
Set it from data:
- Collect around 200 pairs of real questions.
- Label each pair as same meaning or different meaning.
- Compute similarities, and choose the threshold that keeps wrong hits near zero.
- Re-check when you change the embedding model.
Keeping it fresh
- Expire entries after a time limit.
- Invalidate entries when the source documents or the prompt change.
- Sample cache hits for review, like any other output, to catch wrong hits.
What to measure
- Hit rate, overall and by scope.
- Wrong-hit rate, from sampled reviews.
- Latency and cost saved per day.
From the floor
In the production pipeline I teach, the API gateway converts the query into a vector and checks a Redis-backed semantic cache first: a hit goes straight back to the user, and a miss goes on to the routing engine. On the way back, the final response populates the cache, so the next similar question can be served from it, and any personal information that was masked earlier is mapped back before the answer leaves.
When a builder asked whether this was prompt caching or semantic caching, my answer was that it's semantic caching on the user's question, based on its meaning and intent, not exact keywords.