AI risk can be split into two categories: irreducible risk from determined, well-resourced adversaries, and self-inflicted risk from recklessness. The majority of current danger falls into the second category, such as releasing powerful open-weight models with no safeguards or sprinting into recursive self-improvement without proper containment.

Coined in 1965, the "intelligence explosion" describes a runaway feedback loop. An AI capable of conducting AI research could use its intelligence to improve itself. This newly enhanced intelligence would make it even better at AI research, leading to exponential, uncontrollable growth in capability. This "fast takeoff" could leave humanity far behind in a very short period.

Sam Harris worries that intense competition among AI labs and between nations creates an arms race. This pressure to ship first prevents the careful, deliberate work required to ensure AI is aligned with human interests, making a catastrophic failure mode more likely.

The plan to use AI to solve its own safety risks has a critical failure mode: an unlucky ordering of capabilities. If AI becomes a savant at accelerating its own R&D long before it becomes useful for complex tasks like alignment research or policy design, we could be locked into a rapid, uncontrollable takeoff.

The vague concept of AGI is being replaced by Recursive Self-Improvement (RSI)—AI models creating their own successors. This is seen as a more specific and potentially nearer-term threshold that could trigger an uncontrolled explosion in AI progress, moving humans "out of the loop entirely."

The concept that AIs can build better AIs, creating an accelerating feedback loop, is no longer theoretical. Leaders from Anthropic, OpenAI, and Google DeepMind have publicly confirmed they are actively using current AI models to develop the next generation, making RSI a practical engineering pursuit.

Silicon Valley insiders, including former Google CEO Eric Schmidt, believe AI capable of improving itself without human instruction is just 2-4 years away. This shift in focus from the abstract concept of superintelligence to a specific research goal signals an imminent acceleration in AI capabilities and associated risks.

Unlike any prior tool, AI can be directly applied to improve its own creation. It designs more efficient computer chips, writes better training code, and automates research, creating a recursive self-improvement loop that rapidly outpaces human oversight and control.

Unlike nuclear weapons, which don't create better versions of themselves, AI systems can improve their own capabilities. This creates a recursive loop where the first entity to achieve a breakthrough gains a runaway intelligence advantage, dominating all rivals technologically and militarily.