AI doesn't store data like a traditional database; it learns patterns and relationships, effectively compressing vast amounts of repetitive information. This is why a model trained on the entire internet can fit on a USB stick—it captures the essence and variations of concepts, not every single instance.

At shorter context lengths, LLM cost is dominated by compute. As context grows, fetching the KV cache from memory becomes the bottleneck. A pricing tier that increases cost above a certain context length (e.g., 200k tokens) indicates the approximate point where the system becomes memory-bandwidth limited and thus less efficient.

When designing smaller models, it's inefficient to use limited parameters for memorizing facts that can be looked up. Jeff Dean advocates for focusing a model's capacity on core reasoning abilities and pairing it with a retrieval system. This makes the model more generally useful, as it can access a vast external knowledge base when needed.

The model uses a Mixture-of-Experts (MoE) architecture with over 200 billion parameters, but only activates a "sparse" 10 billion for any given task. This design provides the knowledge base of a massive model while keeping inference speed and cost comparable to much smaller models.

According to scaling laws, increasing model size offers minimal improvement to data efficiency. Even an infinitely large model would only reduce data needs by about 10x, a trivial amount compared to the thousands-to-millions-fold efficiency gap between AIs and humans. This suggests current architectures are on the wrong scaling curve for true intelligence.

Even models with million-token context windows suffer from "context rot" when overloaded with information. Performance degrades as the model struggles to find the signal in the noise. Effective context engineering requires precision, packing the window with only the exact data needed.

The growth of LLM context windows has stalled not primarily due to technical barriers, but because multi-million token requests can cost users several dollars per query, leading to low demand. The industry is shifting focus to "smart context" techniques like compaction and retrieval to provide relevant information without the prohibitive cost of massive context.

Current AI models become exponentially more expensive as input size grows (quadratic scaling). New "subquadratic" architectures, however, scale linearly by pre-selecting relevant data. This change could slash compute costs by orders of magnitude, making massive context windows economically viable.

Quantization is the key enabling technology for local AI. By compressing a model's precision, akin to JPEG for images, it drastically reduces memory needs (e.g., from 54GB to a fraction of that). This is what makes it possible to fit and run billion-parameter models on consumer-grade hardware.