Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]

A core cultural tenet at NVIDIA is chasing the "speed of light"—pushing hardware to its absolute theoretical performance limit. This relentless focus on squeezing every ounce of capability out of their silicon is deeply ingrained in their engineering culture and is a key driver of their market leadership.

NVIDIA doesn't simply sell its scarce chips to the highest bidder. It strategically allocates them to cultivate a diverse ecosystem of cloud providers and customers. This prevents any single customer from becoming too powerful and ensures healthy competition among its buyers, which ultimately drives more demand for NVIDIA's hardware.

The security of software is increasingly determined by how much AI compute was spent trying to break it. Companies use diverse AI agents to autonomously attack their own code. The amount of money spent on APIs from labs like Anthropic to find vulnerabilities is becoming the best indicator of a system's robustness.

Optimizing a GPU for low-latency (fast, individual responses) inherently sacrifices its peak throughput (total work done over time). The market's focus on chatbots has over-indexed on latency, creating a major opportunity for companies that build systems optimized for high-throughput, background tasks.

To achieve radical cost reduction, the strategy is to "scavenge" what others won't use: less popular chips (non-NVIDIA), stranded power from intermittent renewables, and small, low-reliability data centers. This arbitrage approach avoids competing for premium resources with deep-pocketed frontier labs.

The need for high-availability data centers is an assumption from the training and real-time era. For asynchronous background agents, a distributed fleet of small, cheap data centers with 95% uptime is viable. Failures are handled by a robust control plane that reroutes work, trading P99 latency for unbeatable economics.

Preventing knowledge transfer from frontier models to open source is impossible. As AI-generated content (code, text) populates the internet, that data becomes part of the training set for the next generation of open-source models. This "latent distillation" ensures a constant diffusion of capabilities.

The "one-time subsidy" of high-quality internet text is largely exhausted. The future of data for training models lies in creating reinforcement learning (RL) "gyms" where agents work on hard, verifiable problems (e.g., coding, math). The environment and the agent's progress become the new data source for recursive self-improvement.

The dominant AI use case will shift from real-time, human-in-the-loop chatbots to long-running background agents. For these agents, which work for hours or days, an extra few seconds of latency is meaningless, unlocking massive cost-saving opportunities by prioritizing throughput over speed.

The market undervalues chips from vendors like AMD because their software stack and kernel libraries are less mature than NVIDIA's CUDA. A team with deep expertise in low-level software and kernel optimization can extract significantly more performance from these chips, creating a powerful arbitrage opportunity by buying them at a discount.

Chips like Cerebras and Grok, which excel at fast memory access (SRAM), are best suited for the compute-bound parts of a transformer model (the MLP). They will likely be paired with traditional GPUs, which are better at handling the memory-capacity demands of the attention mechanism and KV Cache, creating powerful hybrid systems.

To create truly proactive, helpful AI assistants (like a better Siri), systems must be willing to spend vast amounts of cheap compute "speculatively" in the background without a direct user prompt or a guaranteed return on that spend. This is only possible when the cost of a token approaches zero.