The relationship between computing power and AI model capability is not linear. According to established 'scaling laws,' a tenfold increase in the compute used for training large language models (LLMs) results in roughly a doubling of the model's capabilities, highlighting the immense resources required for incremental progress.

At shorter context lengths, LLM cost is dominated by compute. As context grows, fetching the KV cache from memory becomes the bottleneck. A pricing tier that increases cost above a certain context length (e.g., 200k tokens) indicates the approximate point where the system becomes memory-bandwidth limited and thus less efficient.

The GPU architecture is economically optimized for slow AI inference, offering a very low cost per token. However, this efficiency plummets when speed is required, as the cost and power per token increase exponentially, creating a market for alternative architectures in high-speed applications.

Despite models advertising million-token context windows, Blitzy's CEO claims effective intelligence rapidly depreciates beyond 100k tokens due to "context pressure." This suggests that solving large-scale problems requires complex system-level orchestration, not just bigger models.

With long context windows, the memory for KV caches of user sessions presents a massive scaling challenge. For a 10-trillion parameter model, the collective context for just 50 concurrent users could require more memory (5+ terabytes) than the model weights themselves, flipping the infrastructure priority from model storage to session storage.

A KV cache for a single Wikipedia article can consume 80GB of HBM, while a 70B model storing the internet's knowledge is only slightly larger (100GB). This highlights the inefficiency of context-window memory and the benefit of compressing that knowledge into model weights.

Even models with million-token context windows suffer from "context rot" when overloaded with information. Performance degrades as the model struggles to find the signal in the noise. Effective context engineering requires precision, packing the window with only the exact data needed.

The growth of LLM context windows has stalled not primarily due to technical barriers, but because multi-million token requests can cost users several dollars per query, leading to low demand. The industry is shifting focus to "smart context" techniques like compaction and retrieval to provide relevant information without the prohibitive cost of massive context.

While training AI models is a compute-bound problem where more flops yield better results, inference (running the model) is memory-bound. Each token generation requires reading all model weights from memory, making memory bandwidth, not raw processing power, the primary performance bottleneck.