The AI supply chain is crunched not just by obvious components like TSMC wafers and HBM memory. A significant, often overlooked bottleneck is rack manufacturing—including high-speed cables, connectors, and even sheet metal—which are "sneaky hard" due to extreme power, heat, and signal integrity demands.

Templar's Sam Dare argues the perceived GPU scarcity is misunderstood. The actual bottleneck is the limited supply of the latest, well-connected GPUs in data centers. His project aims to create algorithms that can effectively utilize the vast, distributed network of consumer-grade and older enterprise GPUs, unlocking a massive new compute resource.

The short range of copper cables is a key driver behind modern data center design. To maintain bandwidth, GPUs are packed into incredibly dense, megawatt racks. These racks are so heavy they require reinforced concrete floors to support their weight, highlighting a physical bottleneck that photonics technology aims to solve.

The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.

While NVIDIA's GPUs have been the primary AI constraint, the bottleneck is now moving to other essential subsystems. Memory, networking interconnects, and power management are emerging as the next critical choke points, signaling a new wave of investment opportunities in the hardware stack beyond core compute.

AI networking is not an evolution of cloud networking but a new paradigm. It's a 'back-end' system designed to connect thousands of GPUs, handling traffic with far greater intensity, durability, and burstiness than the 'front-end' networks serving general-purpose cloud workloads, requiring different metrics and parameters.

Mixture-of-Experts (MoE) models require an "all-to-all" communication pattern. This is efficient within a single GPU rack's high-speed interconnect but becomes a major bottleneck between racks, where communication is ~8x slower. This effectively limits an MoE layer's maximum size to what a single rack can support.

The key advantage of larger GPU clusters is their ability to use the memory bandwidth of all GPUs in parallel to load model weights. This massive aggregate bandwidth dramatically reduces memory fetch times, which is a primary latency bottleneck, especially for very large, sparse models.

Crusoe Cloud's CEO warns of an impending power density crisis. Today's racks are ~130kW, but NVIDIA's future "Vera Rubin Ultra" chips will demand 600kW per rack—the power of a small town. This massive leap will necessitate fundamental changes in cooling and electrical engineering for all AI infrastructure.