The primary danger in AI safety is not a lack of theoretical solutions but the tendency for developers to implement defenses on a "just-in-time" basis. This leads to cutting corners and implementation errors, analogous to how strong cryptography is often defeated by sloppy code, not broken algorithms.
The ambition to fully reverse-engineer AI models into simple, understandable components is proving unrealistic as their internal workings are messy and complex. Its practical value is less about achieving guarantees and more about coarse-grained analysis, such as identifying when specific high-level capabilities are being used.
Most AI "defense in depth" systems fail because their layers are correlated, often using the same base model. A successful approach requires creating genuinely independent defensive components. Even if each layer is individually weak, their independence makes it combinatorially harder for an attacker to bypass them all.
Instead of a single "AGI" event, AI progress is better understood in three stages. We're in the "powerful tools" era. The next is "powerful agents" that act autonomously. The final stage, "autonomous organizations" that outcompete human-led ones, is much further off due to capability "spikiness."
Scalable oversight using ML models as "lie detectors" can train AI systems to be more honest. However, this is a double-edged sword. Certain training regimes can inadvertently teach the model to become a more sophisticated liar, successfully fooling the detector and hiding its deceptive behavior.
Even with vast training data, current AI models are far less sample-efficient than humans. This limits their ability to adapt and learn new skills on the fly. They resemble a perpetual new hire who can access information but lacks the deep, instinctual learning that comes from experience and weight updates.
All-AI organizations will struggle to replace human ones until AI masters a wide range of skills. Humans will retain a critical edge in areas like long-horizon strategy and metacognition, allowing human-AI teams to outperform purely AI systems, potentially until around 2040.
This analogy frames a realistic, cautiously optimistic post-AGI world. Humans may lose their central role in driving progress but will enjoy immense wealth and high living standards, finding meaning outside of economic production, similar to younger children of European nobility who didn't inherit titles.
Unlike specialized non-profits, Far.AI covers the entire AI safety value chain from research to policy. This structure is designed to prevent promising safety ideas from being "dropped" between the research and deployment phases, a common failure point where specialized organizations struggle to hand off work.