Comparing superintelligence to nuclear weapons is flawed. Nuclear weapons are tools requiring human actors to be deployed. Superintelligence will be an autonomous agent making its own decisions. You can't ensure safety by controlling a human "dictator," because the agent itself operates independently.
Unlike past industrial revolutions where displaced workers could retrain, AI is a 'meta-invention' capable of performing any new task that arises. This eliminates the 'retrain for a new career' safety net, creating a scenario with no Plan B for human employment.
In a world with mass AI-driven unemployment, the economic challenge of providing for everyone's needs is simple due to massive wealth creation. The far more difficult problem is societal: how will humans find meaning and purpose when their jobs, a primary source of identity, are gone?
Developing superintelligence is humanity's top priority. If achieved safely, it can solve other existential risks like climate change. If developed unsafely, it will dominate all other threats, making them irrelevant. In either scenario, superintelligence is the pivotal challenge that dictates the outcome of all others.
The cognitive gap between humans and a future superintelligence will be vast, similar to the gap between a human and their dog. We can't predict its actions because it will operate on a level of abstraction we can't comprehend, just as a dog can't understand why its owner records a podcast. This makes true prediction impossible.
We don't fully understand how advanced AI models work. Creators don't program them with explicit knowledge but train them on vast datasets and then run experiments to discover their capabilities. This makes AI development more of a science—studying an unpredictable artifact—than traditional engineering, highlighting an inherent lack of control.