Coined in 1965, the "intelligence explosion" describes a runaway feedback loop. An AI capable of conducting AI research could use its intelligence to improve itself. This newly enhanced intelligence would make it even better at AI research, leading to exponential, uncontrollable growth in capability. This "fast takeoff" could leave humanity far behind in a very short period.

AI research involves exploring a dependency graph where ideas may fail (stochastic). This contrasts with software engineering's more deterministic path. Success requires "research taste"—an intuition for navigating this uncertainty, a skill often honed in PhD programs.

AI agents have become proficient at following a pre-defined strategy to execute tasks. The next major frontier, and a significant bottleneck, is the ability to explore open-ended environments and generate novel strategies independently. This is the core capability that benchmarks like ARC AGI v3 are designed to test.

The most transformative aspect of AI may be its ability to automate its own research and development. This creates a recursive improvement cycle—an "intelligence explosion"—where progress accelerates exponentially, compressing decades of innovation into a much shorter period.

AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

Anthropic's own launch documents for Mythos and Fable distinguish between engineering and research. While the models significantly accelerate engineering execution (e.g., coding), they have not yet demonstrated the ability to produce novel research insights or judgment. This suggests AI-driven scientific discovery remains a future milestone.

Issues like 'saturation' and 'maxing' reveal a fundamental flaw: benchmarks test narrow, siloed abilities ('Task AGI'). They fail to measure an AI's capacity to combine skills to solve multi-step problems, which is the true bottleneck preventing real-world agentic performance and the next frontier of AI.

Current LLM agents are effective at executing and optimizing experiments within a defined research track, like hyperparameter tuning. However, they lack the crucial scientific skill of 'lateral thinking'—recognizing when a research path is a dead end and strategically pivoting to a fundamentally new approach.

The ultimate goal for leading labs isn't just creating AGI, but automating the process of AI research itself. By replacing human researchers with millions of "AI researchers," they aim to trigger a "fast takeoff" or recursive self-improvement. This makes automating high-level programming a key strategic milestone.