- Essays··6 min read
You can buy GPUs but not memory Memory 🙈
The constraint on AI infrastructure in 2026 is not compute but memory bandwidth adjacent to the silicon, and the CoWoS packaging required to deliver it. TSMC's advanced packaging capacity is fully booked through 2027. Adding GPUs does not reduce LLM inference latency when the bottleneck is HBM bandwidth.
ai-infrastructuregpu-memoryhbmhardware-supply-chainRead - Essays··7 min read
Pulling Apart the Inference Stack
By mid-2026 every serious inference framework has accepted that the two halves of a forward pass want different hardware: prefill on compute-bound GPUs, decode on bandwidth-bound ones, the KV cache shipped between them over a fast fabric. It is the deepest reshaping of LLM serving since continuous batching, and it happened almost entirely without anyone outside the inference crowd noticing.
llm-inferenceprefill-decode-disaggregationvllmkv-cacheRead - Essays··8 min read
Cheap hits, confident wrong answers
Prefix caching is deterministic; semantic caching is probabilistic. One is free and lossless, the other can return a confident, well-formatted, wrong answer with an HTTP 200. Both are true in the same architecture diagram.
llm-inferencesemantic-cachingfinopsprefix-cachingRead