The need for low-latency services for agents and real-time applications in finance and healthcare is driving a shift towards distributed data centers. Instead of remote gigawatt facilities, companies are deploying smaller, power-efficient, air-cooled racks like SambaNova's in existing metropolitan data centers, closer to users.

Traditional web applications concentrated in hubs like Northern Virginia because data transit time dictated latency. Chase Lochmiller notes that for AI, neural network compute time inside the facility vastly overshadows network transit time. Consequently, AI data centers do not need centralized locations and can instead be distributed to regions with low-cost, abundant power.

While AI inference can be decentralized, training the most powerful models demands extreme centralization of compute. The necessity for high-bandwidth, low-latency communication between GPUs means the best models are trained by concentrating hardware in the smallest possible physical space, a direct contradiction to decentralized ideals.

Simply "scaling up" (adding more GPUs to one model instance) hits a performance ceiling due to hardware and algorithmic limits. True large-scale inference requires "scaling out" (duplicating instances), creating a new systems problem of managing and optimizing across a distributed fleet.

The current focus on building massive, centralized AI training clusters represents the 'mainframe' era of AI. The next three years will see a shift toward a distributed model, similar to computing's move from mainframes to PCs. This involves pushing smaller, efficient inference models out to a wide array of devices.

While AI training requires massive, centralized data centers, the growth of inference workloads is creating a need for a new architecture. This involves smaller (e.g., 5 megawatt), decentralized clusters located closer to users to reduce latency. This shift impacts everything from data center design to the software required to manage these distributed fleets.

The initial assumption of a centralized AI model (large hub, large spoke) is wrong. The new model will involve large foundational hubs, enterprise-specific training hubs, and distributed "spokes" of on-premise hardware for inference. This shift is driven by the need for data control and cost efficiency.

While the idea of distributed compute pools is appealing, it's not feasible for AI training due to high latency demands; GPUs must be physically co-located. However, AI inference is less sensitive to this lag, making a distributed network of compute (like home GPUs) a much more viable and exciting model.

Standard inference tooling is designed for one large model on many GPUs. Efficiently serving multiple small models requires the opposite architecture: packing many models onto a single GPU with fast switching to avoid paying for idle hardware, a fundamentally different infrastructure problem.