Essays··8 min read
Semantic Cache Hit, User Miss
OpenAI enabled automatic prompt caching on 1 October 2024, delivering up to 80% latency reduction and 50% cost savings for long prompts. Anthropic launched its version in August 2024, charging $3.75 per million write tokens but only $0.30 per million read tokens, a structure that promised 90% savings on repeated context. Both announcements landed …
semanticcachehituser
Read