While AI inference can be decentralized, training the most powerful models demands extreme centralization of compute. The necessity for high-bandwidth, low-latency communication between GPUs means the best models are trained by concentrating hardware in the smallest possible physical space, a direct contradiction to decentralized ideals.
Related Insights
The 2012 breakthrough that ignited the modern AI era used the ImageNet dataset, a novel neural network, and only two NVIDIA gaming GPUs. This demonstrates that foundational progress can stem from clever architecture and the right data, not just massive initial compute power, a lesson often lost in today's scale-focused environment.
The progress in deep learning, from AlexNet's GPU leap to today's massive models, is best understood as a history of scaling compute. This scaling, resulting in a million-fold increase in power, enabled the transition from text to more data-intensive modalities like vision and spatial intelligence.
The progression from early neural networks to today's massive models is fundamentally driven by the exponential increase in available computational power, from the initial move to GPUs to today's million-fold increases in training capacity on a single model.
The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.
Top-tier kernels like FlashAttention are co-designed with specific hardware (e.g., H100). This tight coupling makes waiting for future GPUs an impractical strategy. The competitive edge comes from maximizing the performance of available hardware now, even if it means rewriting kernels for each new generation.
Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.
According to Stanford's Fei-Fei Li, the central challenge facing academic AI isn't the rise of closed, proprietary models. The more pressing issue is a severe imbalance in resources, particularly compute, which cripples academia's ability to conduct its unique mission of foundational, exploratory research.
Instead of relying on hyped benchmarks, the truest measure of the AI industry's progress is the physical build-out of data centers. Tracking permits, power consumption, and satellite imagery reveals the concrete, multi-billion dollar bets being placed, offering a grounded view that challenges both extreme skeptics and believers.
Today's transformers are optimized for matrix multiplication (MatMul) on GPUs. However, as compute scales to distributed clusters, MatMul may not be the most efficient primitive. Future AI architectures could be drastically different, built on new primitives better suited for large-scale, distributed hardware.
Cohere intentionally designs its enterprise models to fit within a two-GPU footprint. This hard constraint aligns with what the enterprise market can realistically deploy and afford, especially for on-premise settings, prioritizing practical adoption over raw scale.
