AI models excel only at tasks they are specifically trained on, leading to a fragmented skill set rather than a universal intelligence. This "jaggedness," driven by the distribution of training data, will persist even as models become more powerful, challenging the notion of a smooth path to general superintelligence.
Unlike coding, most real-world tasks lack training data that represents the task's actual execution. AIs are trained on descriptions and commentary, not performance data, akin to learning chess from analysis rather than gameplay. This severely limits their practical abilities in most domains.
Even a perfect simulator cannot generate the data needed for superintelligence because crucial data is generated only when real market actors reveal their preferences through behavior. This Hayekian insight suggests this information doesn't exist *prior* to real-world deployment and therefore cannot be simulated.
When an AI improves itself based solely on internal benchmarks (evals), it optimizes for the test, not for real-world utility. This leads to a "Goodhart Singularity," where the AI appears superintelligent on paper but its capabilities fail to generalize outside the lab. The true measure of success is the messy, unpredictable market.
If real-world deployment data is the true bottleneck for AGI, then organizations with unique, proprietary data on economic activity hold far more leverage than they realize. This suggests a power shift from AI labs that build frontier models to the entities—like governments or industries—that control the "signal" from deployment.