Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Tom Milton demonstrates an inductive loop around a GPU power supply feeding an oscilloscope. A forward pass, backward pass, and break are visibly distinct; tiny spikes at each layer even reveal a 12-layer model. At data-center scale, synchronous training produces large, canonical power swings.

Related Insights

Contrary to the conventional view, the training phase for advanced AI models, especially those using reinforcement learning, now demands more compute resources for inference tasks than for the actual backpropagation process where the model learns.

Unlike simple classification (one pass), generative AI performs recursive inference. Each new token (word, pixel) requires a full pass through the model, turning a single prompt into a series of demanding computations. This makes inference a major, ongoing driver of GPU demand, rivaling training.

Tom Milton describes Emoto's prototype on about 32 H100/H200 GPUs: a network tap copies traffic to a verifier node, which reruns the claimed inference and checks outputs match. Unexplained packets reveal hidden workloads like training. This avoids inspecting GPU instructions directly.

The necessity of batching stems from a fundamental hardware reality: moving data is far more energy-intensive than computing with it. A single parameter's journey from on-chip SRAM to the multiplier can cost 1000x more energy than the multiplication itself. Batching amortizes this high data movement cost over many computations.

While AI inference can be decentralized, training the most powerful models demands extreme centralization of compute. The necessity for high-bandwidth, low-latency communication between GPUs means the best models are trained by concentrating hardware in the smallest possible physical space, a direct contradiction to decentralized ideals.

RL models can be inefficient during inference. The GPU often sits idle while the CPU calculates rewards, then suddenly gets hit with a massive "burst" of activity. This unpredictable demand makes serving these models costly and complex, requiring conservative GPU allocation.

While training AI models is a compute-bound problem where more flops yield better results, inference (running the model) is memory-bound. Each token generation requires reading all model weights from memory, making memory bandwidth, not raw processing power, the primary performance bottleneck.

The mean time between failure for GPUs depends heavily on their use. Chips used for intensive model training have much higher failure rates than those used for inference. Investors often conflate the two, underestimating the high churn and replacement costs for training hardware.

While the idea of distributed compute pools is appealing, it's not feasible for AI training due to high latency demands; GPUs must be physically co-located. However, AI inference is less sensitive to this lag, making a distributed network of compute (like home GPUs) a much more viable and exciting model.

GPUs are ill-suited for generating output tokens because they process models in discrete chunks called "kernels." This method requires constant data transfer to and from external memory, creating a significant bottleneck limited by memory bandwidth, which ultimately slows down real-time inference.

Power fluctuations reveal whether GPUs run inference or training | RiffOn