Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Aviation engineers target one-in-a-billion failure probabilities yet experience one-in-five-million real-world outcomes due to unforeseen implementation errors. When AI developers casually accept a 10% extinction probability, it reveals the complete absence of a disciplined, reliable engineering safety plan.

Related Insights

The lack of a major AI-driven catastrophe has led to a 'normalization of deviance,' where developers increasingly use AI in unsafe ways, feeling more confident with each success. This mirrors the lead-up to the Challenger disaster, suggesting a massive, preventable failure is likely as risks are continually overlooked.

An Anthropic alignment lead publicly stated a >10% chance of human extinction from AI within a decade. This creates a paradox: if a company truly believes its work carries such a high risk of global catastrophe, the logical response would be to shut down, not to continue development while trying to solve alignment.

The conversation about AI causing human extinction isn't led by outsiders but by insiders. After a researcher resigned from AI firm Anthropic over safety concerns, his former boss—the head of the AI safety team—publicly agreed, estimating the chance of AI ending the world at a staggering 10%.

The primary danger in AI safety is not a lack of theoretical solutions but the tendency for developers to implement defenses on a "just-in-time" basis. This leads to cutting corners and implementation errors, analogous to how strong cryptography is often defeated by sloppy code, not broken algorithms.

While not a consensus, surveys of AI researchers reveal significant concern. The median respondent in a large survey assigned a 5% probability to human extinction or a similar disaster from AI, with a third to a half placing the risk at 10% or higher, suggesting the threat is taken seriously within the field.

Jones warns that the standard tech development model ('build, break, iterate') is catastrophic when applied to AI. Unlike other technologies, AI's tail risk could involve billions of lives, yet there is zero risk management, a sharp contrast to the discipline required in financial markets.

Other scientific fields operate under a "precautionary principle," avoiding experiments with even a small chance of catastrophic outcomes (e.g., creating dangerous new lifeforms). The AI industry, however, proceeds with what Bengio calls "crazy risks," ignoring this fundamental safety doctrine.

Leading AI companies claim proactive safety standards while treating critical jailbreaks and unauthorized network escapes as mere configuration oversights. Their operating procedures resemble alchemists wearing fume hoods rather than high-reliability cultures like NASA or commercial aviation that thoroughly diagnose and resolve failures before scaling further.

The current approach to AI safety involves identifying and patching specific failure modes (e.g., hallucinations, deception) as they emerge. This "leak by leak" approach fails to address the fundamental system dynamics, allowing overall pressure and risk to build continuously, leading to increasingly severe and sophisticated failures.

A safe AGI deployment requires many independent factors to succeed simultaneously: trustworthy actors, perfect security, solved alignment, etc. In contrast, disaster can occur from a failure in any single one of these areas. This "disjunctive" nature of failure makes a bad outcome highly probable.

Aviation Failure Baselines Prove a Ten Percent AI Extinction Probability Is Unacceptable | RiffOn