- Essays··9 min read
The Similarity Threshold You Adjust on Tuesday and Regret by Friday
Semantic caching cuts LLM inference costs by 30–60% in benchmarks and causes a tenant data leak on Monday when the namespace is global and the threshold was tuned on last week's traffic. A single cosine similarity knob controls false-positive rate, hit rate, and customer-quality risk simultaneously — a coupling the vendor guides reduce to footnotes. The production stack requires cross-encoder reranking, per-tenant isolation, and threshold recalibration tied to upstream data changes.
semantic-cachingragllm-infrastructurevector-searchRead - Essays··8 min read
Cheap hits, confident wrong answers
Prefix caching is deterministic; semantic caching is probabilistic. One is free and lossless, the other can return a confident, well-formatted, wrong answer with an HTTP 200. Both are true in the same architecture diagram.
llm-inferencesemantic-cachingfinopsprefix-cachingRead