The moment AI reaches human-level cognitive ability, it effectively surpasses it. This is due to advantages like running 50x faster, perfect memory, and instantly propagating learnings across all copies, creating a superhuman collective intelligence overnight.
"S-risks" are scenarios of astronomical suffering which could be a worse fate than simple annihilation. An example is an AI system creating vast simulated suffering as a blackmail threat, a profoundly non-obvious existential risk that demands research.
The AI alignment field has moved past theory and into an empirical phase. The main bottleneck is now a lack of skilled AI engineers to conduct concrete experiments, red-teaming, and interpretability studies, creating a direct entry path for technical talent.
Even with perfectly aligned AI, humanity faces "gradual disempowerment." In a future economy, humans are resource-intensive. Nations that prioritize efficient AI workers over human populations could gain an insurmountable economic advantage, marginalizing humanity through normal competitive pressures.
While AI alignment gets attention (~1,000 researchers), the risk of a single entity (a company or state) using AI for irreversible power concentration has far fewer people working on it (~20). This makes it one of the most highly-leveraged areas for impact.
The race to AGI could trigger existential events within the next decade, demanding immediate attention. In contrast, climate change's worst impacts are decades away and less likely to cause total human extinction. Furthermore, AGI could accelerate solutions to climate change.
Common advice to "follow your passion" is backward. Developing valuable skills and applying them to help others is what generates lasting fulfillment and passion. Chasing pre-existing interests often leads to hyper-competitive and ultimately unsatisfying fields.
While interest in AI safety has grown, it's dwarfed by the explosion in AI capabilities research. There are only about 1,000 people in technical AI safety versus up to a million working to accelerate AI capabilities, creating a massive talent imbalance on a critical issue.
AI tools are not just replacing tasks; they are massively amplifying individual and small-team productivity. A team of five can now achieve what once took 30 people, making how you direct your efforts in the next few years more consequential than ever before.
Structure your career with a Plan A (your goal), Plan B (alternatives), and a Plan Z (worst-case backup). Defining Plan Z makes taking risks on Plan A feel safer by concretely showing that failure isn't catastrophic and you have a fallback.
In the AI era, job security doesn't come from a fixed skill set. The most valuable work will always be at the frontier of what AI *cannot* yet do. The winning strategy is to continually identify these moving bottlenecks and shift your skills to address them.
The greatest opportunities for impact lie in problems that are overlooked relative to their scale. The fewer people working on a significant issue, the more "low-hanging fruit" is available for a newcomer to make a substantial difference.