The common analogy of AI to electricity is dangerously rosy. AI is more like fire: a transformative tool that, if mismanaged or weaponized, can spread uncontrollably with devastating consequences. This mental model better prepares us for AI's inherent risks and accelerating power.

The debate pitting AI safety against AI opportunity presents a false choice. Historical parallels, like the railroad industry, show that safety regulations (e.g., standardized tracks, air brakes) were essential for enabling greater speed, reliability, and economic potential. Trustworthy AI will unlock greater opportunity.

Unlike a plague or asteroid, the existential threat of AI is 'entertaining' and 'interesting to think about.' This, combined with its immense potential upside, makes it psychologically difficult to maintain the rational level of concern warranted by the high-risk probabilities cited by its own creators.

The debate around AI's impact presents an asymmetric risk. Underestimating AI's capabilities could lead to obsolescence for individuals and companies. Conversely, overestimating its short-term impact results in some wasted preparation, a far less severe and more recoverable outcome.

AI offers incredible short-term benefits, from fixing daily problems to curing diseases. This immediate positive reinforcement makes it extremely difficult for society to acknowledge and address the simultaneous development of long-term, catastrophic risks, creating a classic devil's bargain.

OpenAI's Boaz Barak advises individuals to treat AI risk like the nuclear threat of the past. While society should worry about tail risks, individuals should focus on the high-probability space where their actions matter, rather than being paralyzed by a small probability of doom.

AI will create negative consequences, like the internet spawned the dark web. However, its potential to solve major problems like disease and energy scarcity makes its development a net positive for society, justifying the risks that must be managed along the way.

Other scientific fields operate under a "precautionary principle," avoiding experiments with even a small chance of catastrophic outcomes (e.g., creating dangerous new lifeforms). The AI industry, however, proceeds with what Bengio calls "crazy risks," ignoring this fundamental safety doctrine.

A key failure mode for using AI to solve AI safety is an 'unlucky' development path where models become superhuman at accelerating AI R&D before becoming proficient at safety research or other defensive tasks. This could create a period where we know an intelligence explosion is imminent but are powerless to use the precursor AIs to prepare for it.