TurboPuffer achieved its massive cost savings by building on slow S3 storage. While this increased write latency by 1000x—unacceptable for transactional systems—it was a perfectly acceptable trade-off for search and AI workloads, which prioritize fast reads over fast writes.

AI workloads are limited by memory bandwidth, not capacity. While commodity DRAM offers more bits per wafer, its bandwidth is over an order of magnitude lower than specialized HBM. This speed difference would starve the GPU's compute cores, making the extra capacity useless and creating a massive performance bottleneck.

Amidst a 48% spike in GPU rental costs, AI companies like Anthropic are shifting heavy enterprise users from flat-rate to usage-based pricing. This move, framed as unblocking power users, is fundamentally a response to the industry-wide compute shortage, directly linking the high cost-to-serve with customer pricing.

APIs charge less for input prompts (prefill) than for generating responses (decode). This is because prefill processes many tokens at once, becoming compute-bound. Decode generates tokens one-by-one, making each step dominated by the high, unamortized cost of memory access. The price difference reflects this efficiency gap.

At shorter context lengths, LLM cost is dominated by compute. As context grows, fetching the KV cache from memory becomes the bottleneck. A pricing tier that increases cost above a certain context length (e.g., 200k tokens) indicates the approximate point where the system becomes memory-bandwidth limited and thus less efficient.

While NVIDIA's GPUs have been the primary AI constraint, the bottleneck is now moving to other essential subsystems. Memory, networking interconnects, and power management are emerging as the next critical choke points, signaling a new wave of investment opportunities in the hardware stack beyond core compute.

For any given hardware, there is a fundamental lower bound on inference latency. This "latency floor" is the time required to load the model's total parameters from memory (e.g., HBM) onto the chip. This process cannot be sped up by reducing batch size or other software tricks.

A key way to improve consumer LLM speed and cost is to cache the results for frequently asked, static questions like "When was OpenAI founded?" This approach, similar to Google's knowledge panels, would provide instant answers for a large cohort of queries without engaging expensive GPU resources for every request.

Unlike standard DRAM where products are standardized, HBM is less of a commodity. The complexity of manufacturing HBM—stacking multiple dice and advanced packaging—allows suppliers to differentiate on technology, yield, and thermal performance, giving them a competitive edge beyond just price.