Entrepreneurial individuals are often biased towards starting their own organization for the satisfaction of building something new. However, joining a high-impact organization that is already working and helping it scale can often be a higher-leverage career move, even if it feels less tangible than founding.
While AI alignment gets attention, the risk of AI concentrating immense power in the hands of a few actors (corporations or states) is arguably more neglected. This could enable unprecedented surveillance or create a single company with the economic power of a nation, posing a distinct and severe threat.
As governments increasingly rely on AI for rapid decision-making, they will need AI advisory systems. A critical gap exists for non-profit or public-good 'AI chief of staff' tools. This prevents a conflict of interest where governments depend on AI built by the very companies they are tasked with monitoring.
Even if AI remains aligned and power isn't dangerously concentrated, humanity could still face gradual disempowerment. In this scenario, humans are simply competed out of the economy and lose agency in a world that becomes unfriendly to them. Currently, few proposals exist to prevent this outcome.
Drawing on Nick Bostrom's 'astronomical waste' argument, the focus should be on mitigating existential risks. While accelerating progress brings a better future sooner (adding one year of utopia), preventing a catastrophe preserves the *entire* potential future, making risk mitigation a far higher-leverage activity.
It's easy to rationalize that the most impactful role is at a prestigious, high-paying frontier AI company. Aspiring safety workers should rigorously question their own biases and motives, as this conclusion is very convenient. Seek out friends who will challenge your reasoning to avoid self-deception.
History shows that most attempts to design a perfect utopia age poorly, often appearing dystopian. A more robust goal, termed 'viatopia' by philosopher Will MacAskill, is to create conditions that improve civilization's ability to navigate to the best possible future, preserving options rather than locking in a specific end-state.
A common argument against an AI development pause is that China wouldn't agree. However, since China is currently behind in the AI race, a pause is strategically beneficial for them as it stops the leader from extending their lead. This counterintuitive point suggests a pause is more geopolitically feasible than often assumed.
Instead of predicting the exact arrival of AGI, focus your career planning on the period where your contributions will be most valuable. This shifts the focus from an abstract date to a practical, personal time horizon for action and skill-building, as advised by 80,000 Hours founder Ben Todd.
While the prospect of AGI shortens career planning horizons, it doesn't eliminate the value of building skills ('career capital'). A simple calculation shows that if you can spend one year to become 20% more effective, that investment pays for itself within five years, making it a rational choice even under moderately accelerated timelines.
AI could accelerate technology to the point where space colonization becomes feasible much sooner than expected. The default path is a 'land grab' where the first actors claim the vast resources of space. This is a highly neglected area where work on legal precedents and governance models today could shape humanity's entire cosmic future.
Instead of betting on a single AI timeline, plan your career across three plausible scenarios: 1) A short, fast-takeoff scenario where AI automates R&D by ~2027. 2) A medium timeline where this takes until the 2030s. 3) A long timeline where the current paradigm plateaus. This portfolio approach makes career strategy more robust.