Redwood Research's Chief Scientist Ryan Greenblatt quantifies existential risk not as a small tail risk, but as a coin toss. He believes on the current trajectory, there is a 50-60% probability that misaligned AIs will seize control, with a significant chance of human extinction following such an event.
The paradox of why AI development accelerates despite stated existential risks is explained by a competitive dynamic. Each major lab, like OpenAI or Anthropic, believes it is the most responsible party to develop the technology first, fearing a less cautious competitor will win. This creates an arms race where slowing down is seen as ceding control to a more dangerous actor.
Many who are dismissive of AI alignment problems aren't denying superintelligence is possible. Instead, they often implicitly use a lower capability threshold for terms like "AGI." They may imagine a system that is good at math and coding but cannot automate more complex strategic tasks, thereby avoiding contemplation of a truly world-altering intelligence.
Once AI systems become proficient at AI R&D, they can trigger a recursive self-improvement loop. This process could radically accelerate progress, potentially achieving an amount of algorithmic advancement in a single year that previously took over a decade. This "intelligence explosion" could rapidly create wildly superhuman systems from a starting point of mere human-level competence.
The moment Artificial General Intelligence (AGI) is achieved, it won't be like creating a new human-level mind. Instead, it will be the integration of many existing narrow AI capabilities, each of which is already vastly superhuman (e.g., in math or logic). Therefore, the first "general" intelligence will immediately possess a powerful, non-human profile of extreme strengths from day one.