Framing AI governance as a short, time-limited 'pause' is a mistake. A better approach is a moratorium that ends not after a set time, but only when society's investment in safety and governance is 'commensurate' with the monumental risk of creating superintelligence, a standard we are currently failing to meet.

Public fear of AI often focuses on dystopian, "Terminator"-like scenarios. The more immediate and realistic threat is Orwellian: governments leveraging AI to surveil, censor, and embed subtle political biases into models to control public discourse and undermine freedom.

A pause on training new, more capable AI models could paradoxically increase risk. It would halt progress at the few, relatively safety-conscious frontier labs, allowing less scrupulous competitors to catch up. Meanwhile, compute stockpiling would continue, making any subsequent capability leap even faster and more dangerous.

Framing an AI development pause as a binary on/off switch is unproductive. A better model is to see it as a redirection of AI labor along a spectrum. Instead of 100% of AI effort going to capability gains, a 'pause' means shifting that effort towards defensive activities like alignment, biodefense, and policy coordination, while potentially still making some capability progress.

The vocabulary of AI safety and regulation (e.g., 'national security threats,' 'autonomy risk') is so ambiguous that a power-hungry government could easily abuse it. Any AI model that refuses government orders, such as for mass surveillance, could be labeled an 'autonomy risk' and shut down, creating a pre-built tool for despotism.

Critics argue that proposals like 'AI 2040,' which advocate for government action to slow AI development, are dangerous. They propose granting governments unprecedented surveillance and control to fight a fictional threat, creating a tangible risk of authoritarian abuse that is worse than the problem it aims to solve.

While the current pause on OpenAI's model release is for national security, it creates a mechanism for government intervention. A future administration could use this same power to block models for political or economic reasons, such as perceived negative impacts on jobs or undesirable social opinions.

AI tools could give the president granular, real-time control over the entire federal bureaucracy. This concept of a 'unitary artificial executive' threatens to centralize immense power, enabling a president to override the independent functions and expertise of civil servants at scale.

Tom Davidson argues that a human-led AI takeover has two distinct "shots on goal": first, a power grab by the tech companies that develop superintelligence, and second, the government co-opting that powerful technology for its own ends.