The most significant danger of autonomous weapons is not a single rogue robot, but the emergent, unpredictable behavior of competing AI systems interacting at machine speed. Similar to algorithmic trading 'flash crashes', these interactions could lead to rapid, uncontrolled conflict escalation without a human referee to intervene.

For the military, the toughest AI adoption challenge isn't on offense, but defense: overcoming institutional resistance to granting AI the autonomy needed to defend networks at machine speed. A human-alert system is too slow, creating a major bureaucratic and command-and-control dilemma.

While fears focus on tactical "killer robots," the more plausible danger is automation bias at the strategic level. Senior leaders, lacking deep technical understanding, might overly trust AI-generated war plans, leading to catastrophic miscalculations about a war's ease or outcome.

Instead of automating decisions, the Pentagon's AI strategy focuses on synthesizing vast amounts of data—assets, weather, potential reactions—to expand a human operator's situational awareness, enabling them to make better, more informed choices.

Public fear focuses on AI hypothetically creating new nuclear weapons. The more immediate danger is militaries trusting highly inaccurate AI systems for critical command and control decisions over existing nuclear arsenals, where even a small error rate could be catastrophic.

The true danger of AI is not a cinematic robot uprising, but a slow erosion of human agency. As we replace CEOs, military strategists, and other decision-makers with more efficient AIs, we gradually cede control to inscrutable systems we don't understand, rendering humanity powerless.

When the White House first proposed a policy against using AI for nuclear launch decisions in 2021, DOD officials found it strange. This highlights the incredible speed at which AI's strategic risks have moved from fringe concerns to central policy debates in just a few years.

Countering the idea that slow, manual processes add valuable friction to warfare decisions, the Pentagon's view is that AI maintains critical checks and balances (rules of engagement, approvals). It only removes the inefficient friction of "hunting and pecking" for data, leading to faster and better-informed decisions.

While China's official doctrine on responsible military AI appears similar to that of the U.S., the real concern stems from its political structure. An autocratic regime's incentive to centralize power by removing human decision-makers could lead it to deploy unsafe AI systems, regardless of official policy.