Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

While datacenters offer massive efficiency gains for short-context batch inference, context sizes exceeding 100k tokens diminish that advantage. Because context dominates memory bandwidth over batch size, running personalized, long-context agent workloads locally becomes competitively cost-effective.

Related Insights

At shorter context lengths, LLM cost is dominated by compute. As context grows, fetching the KV cache from memory becomes the bottleneck. A pricing tier that increases cost above a certain context length (e.g., 200k tokens) indicates the approximate point where the system becomes memory-bandwidth limited and thus less efficient.

Despite models advertising million-token context windows, Blitzy's CEO claims effective intelligence rapidly depreciates beyond 100k tokens due to "context pressure." This suggests that solving large-scale problems requires complex system-level orchestration, not just bigger models.

With long context windows, the memory for KV caches of user sessions presents a massive scaling challenge. For a 10-trillion parameter model, the collective context for just 50 concurrent users could require more memory (5+ terabytes) than the model weights themselves, flipping the infrastructure priority from model storage to session storage.

GPU power consumption remains nearly constant regardless of context size, but throughput collapses as the memory required per token grows. This means longer context windows are exponentially less energy-efficient, measured in tokens per watt. This is called the "1-W law."

Even models with million-token context windows suffer from "context rot" when overloaded with information. Performance degrades as the model struggles to find the signal in the noise. Effective context engineering requires precision, packing the window with only the exact data needed.

While training AI models is a compute-bound problem where more flops yield better results, inference (running the model) is memory-bound. Each token generation requires reading all model weights from memory, making memory bandwidth, not raw processing power, the primary performance bottleneck.

Current AI models become exponentially more expensive as input size grows (quadratic scaling). New "subquadratic" architectures, however, scale linearly by pre-selecting relevant data. This change could slash compute costs by orders of magnitude, making massive context windows economically viable.

Achieving huge context lengths isn't just about better algorithms; it's about hardware-model co-design. Models like Kimi from Moonshot AI strategically trade components, like reducing attention heads in favor of more experts, to optimize performance for specific compute and memory constraints.

The evolution of AI towards complex, autonomous "agents" makes relying solely on the cloud slow and expensive, as users burn through token budgets. Nvidia's bet is that running these agents locally on powerful new PC chips will be faster and cheaper for consumers, driving a major hardware shift away from pure cloud computing.

LLM agents often resend their entire history with each action, causing context to grow continuously. This pushes them into the least energy-efficient operating zones, where tokens-per-watt halves with each context doubling, making them a worst-case scenario for power consumption.