If society gets an early warning of an intelligence explosion, the primary strategy should be to redirect the nascent superintelligent AI 'labor' away from accelerating AI capabilities. Instead, this powerful new resource should be immediately tasked with solving the safety, alignment, and defense problems that it creates, such as patching vulnerabilities or designing biodefenses.
The threat of a misaligned, power-seeking AI extends beyond it undermining alignment research. Such an AI would also have strong incentives to sabotage any effort that strengthens humanity's overall position, including biodefense, cybersecurity, or even tools to improve human rationality, as these would make a potential takeover more difficult.
The most likely reason AI companies will fail to implement their 'use AI for safety' plans is not that the technical problems are unsolvable. Rather, it's that intense competitive pressure will disincentivize them from redirecting significant compute resources away from capability acceleration toward safety, especially without robust, pre-agreed commitments.
The core value of the Effective Altruism (EA) community may be its function as an 'engine' for incubating important but non-prestigious, speculative cause areas like AI safety or digital sentience. It provides a community and methodology for tackling problems when the methodology isn't firm and the work is too unconventional for mainstream institutions.
A major disconnect exists: many VCs believe AGI is near but expect moderate societal change, similar to the last 25 years. In contrast, AI safety futurists believe true AGI will cause a radical transformation comparable to the shift from the hunter-gatherer era to today, all within a few decades.
Reporting AI risks only to a small government body is insufficient because it fails to create 'common knowledge.' Public disclosure allows a wide range of experts, including skeptics, to analyze the data and potentially change their minds publicly. This broad, society-wide conversation is necessary to build the consensus needed for costly or drastic policy interventions.
A key failure mode for using AI to solve AI safety is an 'unlucky' development path where models become superhuman at accelerating AI R&D before becoming proficient at safety research or other defensive tasks. This could create a period where we know an intelligence explosion is imminent but are powerless to use the precursor AIs to prepare for it.
The micro-environment of a job—specifically your direct manager and the daily rhythm of work—has a greater impact on satisfaction and productivity than high-level alignment with an organization's mission. Under-rating these mundane, local factors in career decisions is a common mistake, as a poor fit can drain motivation regardless of shared goals.
The most critical feedback loop for an intelligence explosion isn't just AI automating AI R&D (software). It's AI automating the entire physical supply chain required to produce more of itself—from raw material extraction to building the factories that fabricate the chips it runs on. This 'full stack' automation is a key milestone for exponential growth.
Framing an AI development pause as a binary on/off switch is unproductive. A better model is to see it as a redirection of AI labor along a spectrum. Instead of 100% of AI effort going to capability gains, a 'pause' means shifting that effort towards defensive activities like alignment, biodefense, and policy coordination, while potentially still making some capability progress.
Economists skeptical of explosive AI growth use a recent 'outside view,' noting that technologies like the internet didn't cause a productivity boom. Proponents of rapid growth use a much longer historical view, showing that growth rates have accelerated over millennia due to feedback loops—a pattern they believe AI will dramatically continue.
AI accelerationists and safety advocates often appear to have opposing goals, but may actually desire a similar 10-20 year transition period. The conflict arises because accelerationists believe the default timeline is 50-100 years and want to speed it up, while safety advocates believe the default is an explosive 1-5 years and want to slow it down.
To provide a true early warning system, AI labs should be required to report their highest internal benchmark scores every quarter. Tying disclosures only to public product releases is insufficient, as a lab could develop dangerously powerful systems for internal use long before releasing a public-facing model, creating a significant and hidden risk.
Non-profit or government groups aiming to use AI for safety face the risk of being priced out of compute during an intelligence explosion. A financial hedge against this is to invest a portion of their portfolio in compute-exposed stocks like NVIDIA. If compute prices skyrocket, the investment gains would help offset the increased cost of accessing AI labor.
Convergence is difficult because both camps in the AI speed debate have a narrative for why the other is wrong. Skeptics believe fast-takeoff proponents are naive storytellers who always underestimate real-world bottlenecks. Proponents believe skeptics generically invoke 'bottlenecks' without providing specific, insurmountable examples, thus failing to engage with the core argument.