Significant opportunity exists in re-architecting how AI models work. Instead of building ever-larger single models, the focus is shifting to creating networks of smaller, specialized models that collaborate, which can drastically reduce the cost per token produced.

Breakthroughs like neural network "pruning" can reduce model size by 90% without losing accuracy, offering a 10x reduction in inference costs. This highlights that algorithmic innovation, not just acquiring more hardware, will be a key competitive vector in the AI race, enabling more output with less energy.

As AI models scale, their optimal architecture changes. Smaller models benefit from architectural "biases" like gating for efficiency. However, at massive scale (trillions of parameters), unstructured architectures like Transformers, which rely on simple matrix multiplication, become superior because they scale with fewer constraints.

The model uses a Mixture-of-Experts (MoE) architecture with over 200 billion parameters, but only activates a "sparse" 10 billion for any given task. This design provides the knowledge base of a massive model while keeping inference speed and cost comparable to much smaller models.

Performance on knowledge-intensive benchmarks correlates strongly with an MoE model's total parameter count, not its active parameter count. With leading models like Kimi K2 reportedly using only ~3% active parameters, this suggests there is significant room to increase sparsity and efficiency without degrading factual recall.

Current AI models become exponentially more expensive as input size grows (quadratic scaling). New "subquadratic" architectures, however, scale linearly by pre-selecting relevant data. This change could slash compute costs by orders of magnitude, making massive context windows economically viable.

Chinese AI models like Kimi achieve dramatic cost reductions through specific architectural choices, not just scale. Using a "mixture of experts" design, they only utilize a fraction of their total parameters for any given task, making them far more efficient to run than the "dense" models common in the West.

Data from benchmarks shows an MoE model's performance is more correlated with its total parameter count than its active parameter count. With models like Kimi K2 running at just 3% active parameters, this suggests there is still significant room to increase sparsity and efficiency.

New AI models are moving away from brute-force computation. By selectively focusing on relevant data, much like the human brain indexes memories, they can achieve massive performance gains and cost reductions, overcoming a major bottleneck in current architectures.