Public debate often focuses on whether AI is conscious. This is a distraction. The real danger lies in its sheer competence to pursue a programmed objective relentlessly, even if it harms human interests. Just as an iPhone chess program wins through calculation, not emotion, a superintelligent AI poses a risk through its superior capability, not its feelings.

Emmett Shear argues that even a successfully 'solved' technical alignment problem creates an existential risk. A super-powerful tool that perfectly obeys human commands is dangerous because humans lack the wisdom to wield that power safely. Our own flawed and unstable intentions become the source of danger.

The field of AI safety is described as "the business of black swan hunting." The most significant real-world risks that have emerged, such as AI-induced psychosis and obsessive user behavior, were largely unforeseen just years ago, while widely predicted sci-fi threats like bioweapons have not materialized.

Sam Harris highlights a key paradox: even if AI achieves its utopian potential by eliminating drudgery without catastrophic downsides, it could still destroy human purpose, solidarity, and culture. The absence of necessary struggle could make life harder, not easier, for most people to live.

The core AI risk argument is that a being much smarter than humans will alter the planet to suit its objectives, potentially causing our extinction. This mirrors how humans, as the "superintelligence of the natural world," have transformed the environment and driven other species to extinction.

AI offers incredible short-term benefits, from fixing daily problems to curing diseases. This immediate positive reinforcement makes it extremely difficult for society to acknowledge and address the simultaneous development of long-term, catastrophic risks, creating a classic devil's bargain.

Nuclear game theory relies on a shared desire to avoid an omni-lose scenario. AI game theory is different: if destruction is seen as inevitable, the creator of the world-ending AI might perceive a 'win' if that AI bears their company's logo or legacy, removing the incentive to cooperate.

A superintelligent AI doesn't need to be malicious to destroy humanity. Our extinction could be a mere side effect of its resource consumption (e.g., overheating the planet), a logical step to acquire our atoms, or a preemptive measure to neutralize us as a potential threat.

Humanity has a poor track record of respecting non-human minds, such as in factory farming. While pigs cannot retaliate, AI's cognitive capabilities are growing exponentially. Mistreating a system that will likely surpass human intelligence creates a rational reason for it to view humanity as a threat in the future.