Templar's Sam Dare argues the perceived GPU scarcity is misunderstood. The actual bottleneck is the limited supply of the latest, well-connected GPUs in data centers. His project aims to create algorithms that can effectively utilize the vast, distributed network of consumer-grade and older enterprise GPUs, unlocking a massive new compute resource.

Spreading a model's layers across multiple GPU racks (pipeline parallelism) is a strategy to overcome memory capacity limits on a single rack. However, for inference, it offers no latency improvement; the total time remains the same. Its sole benefit is in memory capacity management for enormous models.

NVIDIA's approach requires connecting thousands of Grok chips, creating latency bottlenecks. Cerebras's CEO argues its single, integrated wafer-scale system avoids this "interconnect tax," offering superior memory bandwidth and performance for massive models by eliminating the wiring between thousands of tiny chips.

Increasing the number of GPUs in a high-speed "scale-up" domain is a physical engineering challenge. It's constrained by the sheer density of cables that can fit within a rack's backplane, along with factors like cable bend radius, power delivery, cooling capacity, and structural weight.

The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.

While AI inference can be decentralized, training the most powerful models demands extreme centralization of compute. The necessity for high-bandwidth, low-latency communication between GPUs means the best models are trained by concentrating hardware in the smallest possible physical space, a direct contradiction to decentralized ideals.

Andrew Feldman, CEO of competitor Cerebras, argues their single wafer-scale chip is superior for large AI models. He contends that connecting thousands of smaller GPUs, as Nvidia does, introduces significant latency from physical wiring that negates on-paper performance specs, creating a fundamental bottleneck.

The public-facing models from major labs are likely efficient Mixture-of-Experts (MOE) versions distilled from much larger, private, and computationally expensive dense models. This means the model users interact with is a smaller, optimized copy, not the original frontier model.

The key advantage of larger GPU clusters is their ability to use the memory bandwidth of all GPUs in parallel to load model weights. This massive aggregate bandwidth dramatically reduces memory fetch times, which is a primary latency bottleneck, especially for very large, sparse models.