# Semantic Caching for LLMs: Cut Cost on Repeat Questions

> Most chatbot traffic is the same handful of questions worded differently. Semantic caching catches those near-duplicates and skips the model call entirely.

URL: https://www.matthewswong.com/en/blog/semantic-caching-llm-cost-reduction/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-07-06
- Updated: 2026-07-06
- Category: AI

Run analytics on any production chatbot for a week and a pattern jumps out: a small set of questions accounts for a huge share of traffic, and almost none of them are worded identically. Users ask what your refund policy is, then what happens if I want my money back, then can I get a refund. Three different strings, one intent, three full model calls billed at three full prices.

Exact-match caching cannot see this. It hashes the literal prompt string, so any change in wording, punctuation, or word order is a cache miss. Semantic caching fixes that by comparing meaning instead of characters, using embedding vectors and a similarity threshold to decide when a stored answer is close enough to reuse. This post walks through how it works, where to set the threshold, how to invalidate stale entries, and when it beats the exact-match prompt caching built into providers like Anthropic and Amazon Bedrock.

## Exact match versus semantic match

Provider-side prompt caching, the kind Anthropic and Amazon Bedrock ship natively, works on a byte-identical prefix. It is extremely effective for long, static context such as a system prompt, a document, or a tool schema that repeats across turns, because the provider skips recomputing the attention state for that exact prefix. It does nothing for the actual user question, because that part of the prompt changes on every request by definition. Semantic caching targets exactly that gap: the variable, natural-language part of the request that provider caches cannot touch.

- Exact-match caching keys on a hash of the literal text, so a single extra space or a rephrased sentence is a full miss.
- Semantic caching keys on an embedding vector, so paraphrases, typos, and reordered words can still register as a hit.
- The two are complementary, not competing: use provider prompt caching for the stable prefix and a semantic cache for the variable question on top of it.

## How the lookup actually works

The mechanics are simple even though the payoff is large. Every incoming query is converted into a fixed-length embedding vector using a small, cheap embedding model, then compared against previously stored query vectors using cosine similarity, which measures the angle between two vectors rather than their raw distance. If the closest stored vector clears a similarity threshold, the cached answer is returned in milliseconds and the request never touches the language model. If nothing clears the bar, the request goes to the model as normal and the new query, its embedding, and the fresh answer get written back into the cache for next time. Under the hood, most production setups do not do a brute-force scan over every stored vector. They rely on an approximate nearest-neighbor index, commonly an HNSW graph or an IVF-based structure, so lookups stay in the low milliseconds even with millions of cached entries. Redis, Postgres with pgvector, and managed vector databases all support this pattern, and the choice of index mostly comes down to what your team already operates rather than raw performance, since the difference only matters at very large scale.

> Store the embedding model version alongside each cache entry. If you ever swap embedding models, old vectors are not comparable to new ones and every entry needs to be recomputed, not just re-read.

## Picking a similarity threshold

The threshold is the single knob that decides whether your cache saves money or serves wrong answers. Set it too low and semantically unrelated questions start returning each other's cached answers, which is worse than a cache miss because the user gets confidently wrong information with no error signal. Set it too high and you barely improve on exact-match caching, because only near-identical phrasing clears the bar. Published implementations cluster in a narrow band: cosine similarity around 0.8 to 0.85 is a common starting point for general Q&A, tightening toward 0.9 or higher for anything where a wrong answer has real consequences, such as pricing, refund eligibility, or medical or legal content.

| Threshold range | Typical use case | Risk profile |
| --- | --- | --- |
| 0.60 to 0.75 | Broad topical grouping, internal search suggestions | High false-hit risk, avoid for direct answers |
| 0.80 to 0.88 | General FAQ and support chatbots | Balanced, needs monitoring and a feedback loop |
| 0.90 and above | Pricing, compliance, medical or legal answers | Low false-hit risk, lower hit rate |

> Never tune the threshold once and forget it. Embedding models drift in behavior across versions, and a threshold calibrated for one model can silently produce false hits after an upgrade. Re-validate against a labeled query set whenever you change the embedding model.

## Cache invalidation, the part everyone skips

A semantic cache that never expires is a liability, not a feature, because the world underneath a cached answer keeps changing even when the question stays the same. Build invalidation in from day one, not as a follow-up task.

- Attach a time-to-live to every entry. Something in the range of a few hours to a day works for most support content; pricing and inventory questions need much shorter windows, sometimes minutes.
- Invalidate by tag or namespace when the underlying source changes. If a policy document updates, purge every cache entry tagged with that document rather than waiting for individual TTLs to lapse.
- Version the cache key space whenever the system prompt, the model, or the embedding model changes, so old entries cannot leak into a new configuration.
- Log every cache hit with its similarity score. A sudden spike in scores clustering right at the threshold line is an early warning that the cache is being stretched too far.

## Where semantic caching pays off, and where it does not

This technique is not a universal upgrade. It shines in specific traffic shapes and can actively hurt others.

- High-value target: customer support and FAQ bots, internal knowledge assistants, and any chatbot fielding the same handful of intents from many different users in many different phrasings.
- Poor fit: creative writing, code generation, or any workload where each request is genuinely novel, because there is nothing to reuse and the embedding lookup adds latency for zero benefit.
- Risky without extra care: anything where a stale or approximately-correct answer causes real harm, such as financial, medical, or legal guidance, unless the threshold is set very conservatively and paired with strict invalidation.

> Teams that report the largest savings almost always pair semantic caching with basic query normalization first, lowercasing, trimming whitespace, and stripping filler words before embedding. That single step often lifts the hit rate more than any threshold tweak.

## A minimal implementation checklist

If you are adding this to an existing chatbot or RAG pipeline, the smallest viable version needs four pieces: an embedding model you already trust, a vector index that supports approximate nearest-neighbor search with a similarity score returned alongside each hit, a threshold you have validated against a labeled set of real queries rather than guessed, and a TTL and invalidation path tied to whatever content the answers depend on. Start with a conservative threshold, measure the hit rate and the false-hit rate separately using logged feedback, and loosen the threshold gradually rather than starting loose and tightening after an incident. Track three numbers from week one: cache hit rate, estimated dollar savings per day based on the model's per-token price, and a manually reviewed sample of hits to catch false positives your automated logging cannot see on its own. Most teams find the first real savings within days, and the biggest additional gains after that come from query normalization and threshold tuning, not from switching embedding models or vector databases.

## Sources

- AWS: Implementing a semantic cache with ElastiCache for Valkey
- AWS Bedrock: Prompt caching for faster model inference
- Anthropic: Prompt caching documentation
- Wikipedia: Cosine similarity

## Source links

- https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/semantic-caching-implementation.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
- https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- https://en.wikipedia.org/wiki/Cosine_similarity

## FAQ

### What is semantic caching for LLMs?

Semantic caching stores an embedding vector alongside each cached question and answer pair, then compares a new query's embedding to those stored vectors using cosine similarity. If a stored query is close enough to the new one, the cached answer is returned instead of calling the model again. This lets differently worded questions with the same meaning share a single cached response.

### How is semantic caching different from prompt caching?

Prompt caching, the kind built into Anthropic and Amazon Bedrock, matches an exact, byte-identical prefix such as a system prompt or a repeated document, and skips recomputing that part of the request. Semantic caching matches by meaning rather than exact text, so it catches paraphrased or reworded user questions that prompt caching cannot see. The two techniques target different parts of a request and work well together.

### How do I choose a similarity threshold for a semantic cache?

Start conservative, around 0.85 to 0.9 cosine similarity, and validate it against a labeled set of real user queries before loosening it. Lower thresholds increase the hit rate but also increase the chance of returning an answer to a question the user did not actually ask. Content where a wrong answer has real consequences, such as pricing or compliance questions, should use a higher threshold than general FAQ traffic.

### Can semantic caching return the wrong answer?

Yes, this is called a false hit and it is the main risk of the technique. It happens when two questions are embedded close together despite having meaningfully different intents, and the threshold is loose enough to treat them as a match. Mitigate it by setting a conservative threshold, logging similarity scores on every hit, and re-validating the threshold whenever the embedding model changes.

### When should I not use semantic caching?

Skip it for workloads where each request is genuinely novel, such as creative writing or code generation, because there is nothing to reuse and the embedding lookup only adds latency. Also be cautious with high-stakes domains like medical or legal advice unless the threshold is set very conservatively and paired with strict cache invalidation, since a stale or approximate answer there can cause real harm.

---

- HTML version: https://www.matthewswong.com/en/blog/semantic-caching-llm-cost-reduction/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/semantic-caching-llm-cost-reduction/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json