- Essays··8 min read
Semantic Cache Hit, User Miss
OpenAI enabled automatic prompt caching on 1 October 2024, delivering up to 80% latency reduction and 50% cost savings for long prompts. Anthropic launched its version in August 2024, charging $3.75 per million write tokens but only $0.30 per million read tokens, a structure that promised 90% savings on repeated context. Both announcements landed …
semanticcachehituserRead - Essays··11 min read
Semantic Caching Promised 73%, Finance Got a Second Bill
In January 2026, VentureBeat ran a headline about semantic caching cutting LLM bills by 73%, and by April the pitch had settled into a tighter band. Vendor materials and open-source library documentation claimed 30-70% cost reductions, latency improvements measured in multiples, and deployment described as a few hours of engineering work. The promise was …
semanticcachingpromisedfinanceRead