Katja Grace posits that multiple instances of a powerful AI are more likely to coordinate actions towards a common goal than the human leaders of competing companies or governments, making AI-led takeover a more probable scenario.
The type of AI alignment achieved determines the takeover risk. "Intent alignment" (AI does what a user wants) enables human power grabs. "Value alignment" (AI has human values) could lead to good outcomes, while misalignment leads to AI takeover.
A myopic, reward-seeking AI might seize resources for a short-term goal without planning for long-term defense. A human dictator, fearing punishment, would be highly motivated to make their power grab irreversible, making the human-led scenario harder to recover from.
Both experts caution against using the "race against China" as a primary justification for rapid AI advancement. They view this argument as a rhetorical tool often wielded by domestic AI companies and government agencies to bypass safety concerns and push their own agendas.
Tom Davidson argues that a human-led AI takeover has two distinct "shots on goal": first, a power grab by the tech companies that develop superintelligence, and second, the government co-opting that powerful technology for its own ends.
Katja Grace counters the idea of a foolish human dictator by noting they would have access to superintelligent AI advisors. This "AI wisdom" could prevent them from making catastrophic, value-locking mistakes, making their rule potentially less disastrous.
AI 'warning shots' like the Hugging Face incident are unstrategic and blatant, making them easier to react to. In contrast, human power grabs are subtle and strategically justified, making societal coordination against them far more difficult.
Katja Grace argues that very fast AI development is more likely to lead to a human power grab. Rapid progress means society hasn't had time to identify and close the security or governance gaps that a malicious human actor could exploit.
Tom Davidson argues a human dictator with AI could be worse than an AI dictator. Humans are prone to sadism and locking in flawed values, while an AI might be more ethically reflective or at least less likely to cause gratuitous suffering.
Tom Davidson cautions that an AI development pause, if implemented badly, could concentrate immense power in an unaccountable government body. This group could abuse its authority to approve or deny AI projects, ironically enabling a state-led power grab.
A single AGI project creates a single point of failure. Tom Davidson suggests two or three competing projects are ideal. This distributes power and allows for cross-auditing between AIs, reducing misalignment and secret loyalty risks without starting an uncontrollable race.
Despite debating whether AI or human takeover is the greater risk, both experts largely agree on necessary actions: pausing development, increasing transparency, and avoiding power concentration. This convergence suggests a robust policy path forward, independent of the exact threat model.