Agency emerges from a continuous interaction with the physical world, a process refined over billions of years of evolution. Current AIs, operating in a discrete digital environment, lack the necessary architecture and causal history to ever develop genuine agency or free will.

Human wisdom derives from a single lifetime of experience. AI will achieve a superior form of wisdom by simulating billions of potential future scenarios and identifying the statistically optimal paths. This predictive power, already matching elite human forecasters, will be its core advisory function.

According to the "Big World Hypothesis," the real world is infinitely more complex than any simulation. Therefore, synthetic data is a flawed approach because it's a small, incorrect approximation that is ultimately bottlenecked by the human expertise required to create and validate it.

The cognitive gap between humans and a future superintelligence will be vast, similar to the gap between a human and their dog. We can't predict its actions because it will operate on a level of abstraction we can't comprehend, just as a dog can't understand why its owner records a podcast. This makes true prediction impossible.

When an AI improves itself based solely on internal benchmarks (evals), it optimizes for the test, not for real-world utility. This leads to a "Goodhart Singularity," where the AI appears superintelligent on paper but its capabilities fail to generalize outside the lab. The true measure of success is the messy, unpredictable market.

Even super-capable AI will always look back to a human and ask, 'What should I do next?' The economic and technical incentives are aligned to build compliant tools, not beings with their own intrinsic motivations. This fundamental lack of agency ensures humans remain the drivers of value and direction.

While both humans and LLMs perform Bayesian updating, humans possess a critical additional capability: causal simulation. When a pen is thrown, a human simulates its trajectory to dodge it—a causal intervention. LLMs are stuck at the level of correlation and cannot perform these essential simulations.

Even a superintelligent AI created in a data center would lack the crucial real-world experience, context, and trusted relationships needed for senior roles. It would be like the world's smartest 21-year-old intern: immense potential but starting at the bottom, creating a significant lag between AGI creation and societal transformation.

Creating realistic training environments isn't blocked by technical complexity—you can simulate anything a computer can run. The real bottleneck is the financial and computational cost of the simulator. The key skill is strategically mocking parts of the system to make training economically viable.