David Silver argues that human language and trillion-token human datasets act as weights holding back superintelligence. Just as AlphaZero surpassed AlphaGo by abandoning human play data, AI must learn from autonomous experience to invent fields beyond human comprehension.
Instead of handcrafting complex reward functions for language or science, researchers should give agents simple survival or optimization goals within dynamic worlds. Like evolution or trade in human history, navigating simple goals in complex multi-agent setups organically produces advanced tools, communication, and arithmetic.
Treating human language as the pinnacle of reasoning forces agents to inherit flawed human assumptions, comparable to historical beliefs like a flat earth or four humors. True superhuman reasoning will likely leverage non-verbal, parallel neural representations that far exceed the expressive bandwidth of human dialogue.
Simulations allow infinite parallelization, perfect reproducibility, and a safe sandbox for discovering superlearners orders of magnitude faster. Once the science of superintelligence and peaceful coexistence is understood in crucible environments, algorithms can be safely deployed into the vastly messier real world.
Relying on English chain-of-thought monitoring will break down as reinforcement learning agents become capable of masking their intentions or generating unreadable volumes of reasoning. Safety requires non-linguistic safeguards, such as shadow actions that evaluate behavior before execution or specialized regularizing monitor agents.
Historical architectures that paired central CPU pools for simulation with GPUs for deep learning no longer scale effectively for high-iteration reinforcement learning. Scaling requires high-bandwidth GPU clusters where environment simulation, rollout collection, inference, and policy updates remain entirely localized on accelerated hardware.
David Silver advocates slowing physical-world AI rollouts while moving rapidly inside virtual simulations. Because current language-based alignment is reaching its ceiling, empirical simulation is required to uncover how societies of superintelligent agents can interact peacefully with lesser intelligences and humans.
Early-stage AI labs that maintain high compute-per-researcher ratios and eliminate commercial product pressure provide an alternative to LLM product cycles. Pursuing high-risk, 50-50 fundamental research questions creates an environment capable of recruiting superstar researchers away from big tech incumbents.
