Caching

Retrieval pipelines repeat a lot of work — the same queries, the same embeddings, the same answers. Caching stores and reuses those results to cut latency and cost dramatically. And the most powerful form — semantic caching — even reuses results for queries that mean the same thing but are worded differently.

💡 In one line: Caching reuses previous embeddings, retrievals, and answers to make repeated (or similar) queries far faster and cheaper.

Why Cache?

Every stage — embedding, retrieval, reranking, and the LLM call — costs time and money. In practice, many queries repeat or are near-duplicates. Caching reuses prior work so you don't pay for it twice.

What to Cache

Caching can happen at several layers of the pipeline:

• Embedding cache — reuse embeddings for repeated text (skip re-embedding).
• Retrieval cache — reuse the top-k results for a query.
• Answer cache — reuse the generated LLM answer.
• Reranker cache — reuse rerank scores for seen query-document pairs.

Whiteboard

Exact vs. Semantic Caching

• Exact-match cache — the key is the query string (hashed). Instant on identical queries, but misses paraphrases.
• Semantic cache — the key is the query's meaning (its embedding). It returns a cached result when a new query is similar enough to a previous one, so it catches paraphrases ("reset my password" vs "I forgot my login").

How Semantic Caching Works

Embed the query, look for a close enough cached query, and reuse its result.

Whiteboard

The Similarity Threshold

The critical knob for a semantic cache:

• Too high → few hits (it misses paraphrases).
• Too low → false hits (it returns the wrong cached answer).

Tune it carefully and monitor. (Tools like GPTCache implement this pattern.)

Cache Invalidation

Caches go stale when the underlying corpus changes. Manage it with:

• TTL (expiry times),
• versioning of the index/data, or
• invalidation on updates.

As the saying goes, "cache invalidation is one of the hard problems" — plan for it.

Multi-Tenancy & Safety

Scope caches per user or tenant so results never leak across boundaries. Avoid caching personalised or sensitive results in a shared, global cache.

Trade-offs

• Benefits — big latency and cost savings, higher throughput.
• Risks — stale answers, semantic false hits, memory use, and invalidation complexity.

Code Example

Best Practices

• Cache at multiple layers (embedding, retrieval, answer).
• Tune the semantic threshold and set sensible TTLs.
• Scope per tenant, and invalidate on data changes.
• Monitor the hit rate to confirm it's actually helping.

Summary

• Caching reuses prior embeddings, retrievals, and answers to cut latency and cost.
• Cache at multiple layers of the pipeline.
• Exact caching matches identical queries; semantic caching matches by meaning.
• A semantic cache hinges on a well-tuned similarity threshold.
• Plan for invalidation (TTL/versioning) and scope caches per tenant for safety. EOF echo created