Run analytics on any production chatbot for a week and a pattern jumps out: a small set of questions accounts for a huge share of traffic, and almost none of them are worded identically. Users ask what your refund policy is, then what happens if I want my money back, then can I get a refund. Three different strings, one intent, three full model calls billed at three full prices.
Exact-match caching cannot see this. It hashes the literal prompt string, so any change in wording, punctuation, or word order is a cache miss. Semantic caching fixes that by comparing meaning instead of characters, using embedding vectors and a similarity threshold to decide when a stored answer is close enough to reuse. This post walks through how it works, where to set the threshold, how to invalidate stale entries, and when it beats the exact-match prompt caching built into providers like Anthropic and Amazon Bedrock.
Exact match versus semantic match
Provider-side prompt caching, the kind Anthropic and Amazon Bedrock ship natively, works on a byte-identical prefix. It is extremely effective for long, static context such as a system prompt, a document, or a tool schema that repeats across turns, because the provider skips recomputing the attention state for that exact prefix. It does nothing for the actual user question, because that part of the prompt changes on every request by definition. Semantic caching targets exactly that gap: the variable, natural-language part of the request that provider caches cannot touch.
- Exact-match caching keys on a hash of the literal text, so a single extra space or a rephrased sentence is a full miss.
- Semantic caching keys on an embedding vector, so paraphrases, typos, and reordered words can still register as a hit.
- The two are complementary, not competing: use provider prompt caching for the stable prefix and a semantic cache for the variable question on top of it.
How the lookup actually works
The mechanics are simple even though the payoff is large. Every incoming query is converted into a fixed-length embedding vector using a small, cheap embedding model, then compared against previously stored query vectors using cosine similarity, which measures the angle between two vectors rather than their raw distance. If the closest stored vector clears a similarity threshold, the cached answer is returned in milliseconds and the request never touches the language model. If nothing clears the bar, the request goes to the model as normal and the new query, its embedding, and the fresh answer get written back into the cache for next time. Under the hood, most production setups do not do a brute-force scan over every stored vector. They rely on an approximate nearest-neighbor index, commonly an HNSW graph or an IVF-based structure, so lookups stay in the low milliseconds even with millions of cached entries. Redis, Postgres with pgvector, and managed vector databases all support this pattern, and the choice of index mostly comes down to what your team already operates rather than raw performance, since the difference only matters at very large scale.
# Read-through semantic cache lookup (pseudocode)
def handle_query(user_message: str) -> dict:
query_vec = embed(user_message) # e.g. 1024-d model
hit = vector_store.search(query_vec, k=3) # cosine distance index
if hit and hit.score >= SIMILARITY_THRESHOLD: # e.g. 0.85
return {"response": hit.answer, "source": "cache"}
answer = call_llm(user_message) # cache miss -> real call
vector_store.upsert(query_vec, answer, ttl=86400)
return {"response": answer, "source": "llm"}
Store the embedding model version alongside each cache entry. If you ever swap embedding models, old vectors are not comparable to new ones and every entry needs to be recomputed, not just re-read.
Picking a similarity threshold
The threshold is the single knob that decides whether your cache saves money or serves wrong answers. Set it too low and semantically unrelated questions start returning each other's cached answers, which is worse than a cache miss because the user gets confidently wrong information with no error signal. Set it too high and you barely improve on exact-match caching, because only near-identical phrasing clears the bar. Published implementations cluster in a narrow band: cosine similarity around 0.8 to 0.85 is a common starting point for general Q&A, tightening toward 0.9 or higher for anything where a wrong answer has real consequences, such as pricing, refund eligibility, or medical or legal content.
| Threshold range | Typical use case | Risk profile |
|---|---|---|
| 0.60 to 0.75 | Broad topical grouping, internal search suggestions | High false-hit risk, avoid for direct answers |
| 0.80 to 0.88 | General FAQ and support chatbots | Balanced, needs monitoring and a feedback loop |
| 0.90 and above | Pricing, compliance, medical or legal answers | Low false-hit risk, lower hit rate |
Never tune the threshold once and forget it. Embedding models drift in behavior across versions, and a threshold calibrated for one model can silently produce false hits after an upgrade. Re-validate against a labeled query set whenever you change the embedding model.
Cache invalidation, the part everyone skips
A semantic cache that never expires is a liability, not a feature, because the world underneath a cached answer keeps changing even when the question stays the same. Build invalidation in from day one, not as a follow-up task.
- Attach a time-to-live to every entry. Something in the range of a few hours to a day works for most support content; pricing and inventory questions need much shorter windows, sometimes minutes.
- Invalidate by tag or namespace when the underlying source changes. If a policy document updates, purge every cache entry tagged with that document rather than waiting for individual TTLs to lapse.
- Version the cache key space whenever the system prompt, the model, or the embedding model changes, so old entries cannot leak into a new configuration.
- Log every cache hit with its similarity score. A sudden spike in scores clustering right at the threshold line is an early warning that the cache is being stretched too far.
Where semantic caching pays off, and where it does not
This technique is not a universal upgrade. It shines in specific traffic shapes and can actively hurt others.
- High-value target: customer support and FAQ bots, internal knowledge assistants, and any chatbot fielding the same handful of intents from many different users in many different phrasings.
- Poor fit: creative writing, code generation, or any workload where each request is genuinely novel, because there is nothing to reuse and the embedding lookup adds latency for zero benefit.
- Risky without extra care: anything where a stale or approximately-correct answer causes real harm, such as financial, medical, or legal guidance, unless the threshold is set very conservatively and paired with strict invalidation.
Teams that report the largest savings almost always pair semantic caching with basic query normalization first, lowercasing, trimming whitespace, and stripping filler words before embedding. That single step often lifts the hit rate more than any threshold tweak.
A minimal implementation checklist
If you are adding this to an existing chatbot or RAG pipeline, the smallest viable version needs four pieces: an embedding model you already trust, a vector index that supports approximate nearest-neighbor search with a similarity score returned alongside each hit, a threshold you have validated against a labeled set of real queries rather than guessed, and a TTL and invalidation path tied to whatever content the answers depend on. Start with a conservative threshold, measure the hit rate and the false-hit rate separately using logged feedback, and loosen the threshold gradually rather than starting loose and tightening after an incident. Track three numbers from week one: cache hit rate, estimated dollar savings per day based on the model's per-token price, and a manually reviewed sample of hits to catch false positives your automated logging cannot see on its own. Most teams find the first real savings within days, and the biggest additional gains after that come from query normalization and threshold tuning, not from switching embedding models or vector databases.