The most transformative aspect of AI may be its ability to automate its own research and development. This creates a recursive improvement cycle—an "intelligence explosion"—where progress accelerates exponentially, compressing decades of innovation into a much shorter period.
Widespread unemployment is a serious concern, but it's a symptom of a much larger shift. Advanced AI automating core cognitive labor could trigger a societal transformation as profound as the agricultural or industrial revolutions, fundamentally reshaping our way of life.
The unique danger of an AI-driven transformation is its unprecedented speed. Unlike past revolutions that unfolded over centuries, AI could reshape society in decades or less, overwhelming our institutions' ability to adapt, similar to how it took 50 years to get the Paris Climate Agreement.
While not a consensus, surveys of AI researchers reveal significant concern. The median respondent in a large survey assigned a 5% probability to human extinction or a similar disaster from AI, with a third to a half placing the risk at 10% or higher, suggesting the threat is taken seriously within the field.
Deciding a career path based on uncertain future events is challenging. The "expected value" framework helps by multiplying the massive potential impact (averting catastrophe) by its probability. Even with a small probability, the expected value of working on AI safety can be enormous, justifying the risk.
The focus on AGI can obscure more immediate threats. Even narrowly capable AI tools pose existential risks. For example, an AI that only excels at biotechnology research could make it easy for malicious actors to develop dangerous pathogens, regardless of its general intelligence.
AI is fundamentally different from past technologies. While tools like the steam engine amplified human physical labor, AI has the potential to replace and reproduce flexible human cognitive labor—the very engine of historical progress and innovation. This makes its potential impact far greater.
While thousands work on AI safety, the field is severely neglected relative to the problem's potential scale. For perspective, the Nature Conservancy alone employs more people (3,000-4,000) than the estimated number working globally on the most severe risks from AGI, highlighting a massive resource disparity.
The argument that AI risk is "too sci-fi" is historically weak. H.G. Wells described atomic bombs in a 1914 novel, decades before they were built and while leading physicists deemed them impossible. What seems like science fiction can become a geopolitical reality.