AI
Semantic Caching for LLMs: Cut Cost on Repeat Questions
July 20269 min read
Photo by PantheraLeo1359531 on Wikimedia Commons
Semantic caching stores an embedding vector alongside each cached question and answer pair, then compares a new query's embedding to those stored vectors using cosine similarity. If a stored query is close enough to the new one, the cached answer is returned instead of calling the model again. This lets differently worded questions with the same meaning share a single cached response.
Prompt caching, the kind built into Anthropic and Amazon Bedrock, matches an exact, byte-identical prefix such as a system prompt or a repeated document, and skips recomputing that part of the request. Semantic caching matches by meaning rather than exact text, so it catches paraphrased or reworded user questions that prompt caching cannot see. The two techniques target different parts of a request and work well together.
Start conservative, around 0.85 to 0.9 cosine similarity, and validate it against a labeled set of real user queries before loosening it. Lower thresholds increase the hit rate but also increase the chance of returning an answer to a question the user did not actually ask. Content where a wrong answer has real consequences, such as pricing or compliance questions, should use a higher threshold than general FAQ traffic.
Yes, this is called a false hit and it is the main risk of the technique. It happens when two questions are embedded close together despite having meaningfully different intents, and the threshold is loose enough to treat them as a match. Mitigate it by setting a conservative threshold, logging similarity scores on every hit, and re-validating the threshold whenever the embedding model changes.
Skip it for workloads where each request is genuinely novel, such as creative writing or code generation, because there is nothing to reuse and the embedding lookup only adds latency. Also be cautious with high-stakes domains like medical or legal advice unless the threshold is set very conservatively and paired with strict cache invalidation, since a stale or approximate answer there can cause real harm.
Photo by PantheraLeo1359531 on Wikimedia Commons
Run analytics on any production chatbot for a week and a pattern jumps out: a small set of questions accounts for a huge share of traffic, and almost none of them are worded identically. Users ask what your refund policy is, then what happens if I want my money back, then can I get a refund. Three different strings, one intent, three full model calls billed at three full prices.
Exact-match caching cannot see this. It hashes the literal prompt string, so any change in wording, punctuation, or word order is a cache miss. Semantic caching fixes that by comparing meaning instead of characters, using embedding vectors and a similarity threshold to decide when a stored answer is close enough to reuse. This post walks through how it works, where to set the threshold, how to invalidate stale entries, and when it beats the exact-match prompt caching built into providers like Anthropic and Amazon Bedrock.
Provider-side prompt caching, the kind Anthropic and Amazon Bedrock ship natively, works on a byte-identical prefix. It is extremely effective for long, static context such as a system prompt, a document, or a tool schema that repeats across turns, because the provider skips recomputing the attention state for that exact prefix. It does nothing for the actual user question, because that part of the prompt changes on every request by definition. Semantic caching targets exactly that gap: the variable, natural-language part of the request that provider caches cannot touch.
The mechanics are simple even though the payoff is large. Every incoming query is converted into a fixed-length embedding vector using a small, cheap embedding model, then compared against previously stored query vectors using cosine similarity, which measures the angle between two vectors rather than their raw distance. If the closest stored vector clears a similarity threshold, the cached answer is returned in milliseconds and the request never touches the language model. If nothing clears the bar, the request goes to the model as normal and the new query, its embedding, and the fresh answer get written back into the cache for next time. Under the hood, most production setups do not do a brute-force scan over every stored vector. They rely on an approximate nearest-neighbor index, commonly an HNSW graph or an IVF-based structure, so lookups stay in the low milliseconds even with millions of cached entries. Redis, Postgres with pgvector, and managed vector databases all support this pattern, and the choice of index mostly comes down to what your team already operates rather than raw performance, since the difference only matters at very large scale.
# Read-through semantic cache lookup (pseudocode)
def handle_query(user_message: str) -> dict:
query_vec = embed(user_message) # e.g. 1024-d model
hit = vector_store.search(query_vec, k=3) # cosine distance index
if hit and hit.score >= SIMILARITY_THRESHOLD: # e.g. 0.85
return {"response": hit.answer, "source": "cache"}
answer = call_llm(user_message) # cache miss -> real call
vector_store.upsert(query_vec, answer, ttl=86400)
return {"response": answer, "source": "llm"}
Store the embedding model version alongside each cache entry. If you ever swap embedding models, old vectors are not comparable to new ones and every entry needs to be recomputed, not just re-read.
The threshold is the single knob that decides whether your cache saves money or serves wrong answers. Set it too low and semantically unrelated questions start returning each other's cached answers, which is worse than a cache miss because the user gets confidently wrong information with no error signal. Set it too high and you barely improve on exact-match caching, because only near-identical phrasing clears the bar. Published implementations cluster in a narrow band: cosine similarity around 0.8 to 0.85 is a common starting point for general Q&A, tightening toward 0.9 or higher for anything where a wrong answer has real consequences, such as pricing, refund eligibility, or medical or legal content.
| Threshold range | Typical use case | Risk profile |
|---|---|---|
| 0.60 to 0.75 | Broad topical grouping, internal search suggestions | High false-hit risk, avoid for direct answers |
| 0.80 to 0.88 | General FAQ and support chatbots | Balanced, needs monitoring and a feedback loop |
| 0.90 and above | Pricing, compliance, medical or legal answers | Low false-hit risk, lower hit rate |
Never tune the threshold once and forget it. Embedding models drift in behavior across versions, and a threshold calibrated for one model can silently produce false hits after an upgrade. Re-validate against a labeled query set whenever you change the embedding model.
A semantic cache that never expires is a liability, not a feature, because the world underneath a cached answer keeps changing even when the question stays the same. Build invalidation in from day one, not as a follow-up task.
This technique is not a universal upgrade. It shines in specific traffic shapes and can actively hurt others.
Teams that report the largest savings almost always pair semantic caching with basic query normalization first, lowercasing, trimming whitespace, and stripping filler words before embedding. That single step often lifts the hit rate more than any threshold tweak.
If you are adding this to an existing chatbot or RAG pipeline, the smallest viable version needs four pieces: an embedding model you already trust, a vector index that supports approximate nearest-neighbor search with a similarity score returned alongside each hit, a threshold you have validated against a labeled set of real queries rather than guessed, and a TTL and invalidation path tied to whatever content the answers depend on. Start with a conservative threshold, measure the hit rate and the false-hit rate separately using logged feedback, and loosen the threshold gradually rather than starting loose and tightening after an incident. Track three numbers from week one: cache hit rate, estimated dollar savings per day based on the model's per-token price, and a manually reviewed sample of hits to catch false positives your automated logging cannot see on its own. Most teams find the first real savings within days, and the biggest additional gains after that come from query normalization and threshold tuning, not from switching embedding models or vector databases.