The primary constraint to solving major AI safety challenges is the lack of experienced founders and leaders who can build and scale organizations. Ideas are plentiful, and funding follows great talent, making the 'founder bottleneck' the key problem to solve.
Halcyon operates with both a nonprofit grant-making arm and a for-profit venture fund. This hybrid structure serves as a 'tool bag' that allows them to use the best financial instrument for each situation, from pre-idea grants to scaling for-profit companies.
Offering grants to talented individuals before they have chosen a specific project allows them to fully explore a new, complex field like AI safety without financial pressure. This attracts top-tier leaders who might otherwise not make the leap.
AIUC creates an AI safety standard and offers an insurance policy exclusively to companies that meet it. This incentivizes companies to adopt higher safety practices to win enterprise customers, who value the de-risking that comes with the standard and insurance.
AI's rapid progress means we don't have the luxury of developing fields like control, verification, and alignment sequentially. Society must build these nascent industries in parallel, each requiring a 'Manhattan Project-sized effort' to keep pace with capabilities.
Any international or inter-company agreements to slow down AI development are impotent without the technology to verify compliance. The 'teeth' of any treaty or law will be technical verification tools, making their development a critical and urgent priority.
It seems counterintuitive to start new organizations if AGI is near. However, the alternative is inaction. The second-best time to start an AI safety company is today, as these entities are needed to build the solutions we'll rely on if we manage to slow down progress.
A near-future problem will be a surplus of philanthropic capital for AI safety but a deficit of high-quality organizations to absorb it. The founders who start organizations today are building the menu of investable options for the massive wave of funding to come.
While deep mission alignment is critical for co-founders and early research heads, it's less important for functional roles like a CFO. Scaling organizations should prioritize competence for these positions, leveraging generalist recruiting firms instead of niche, mission-focused searches.
Unlike the novel and abstract challenge of AI alignment, biosecurity threats are known problems. Solutions, like stockpiling PPE, are concrete and can be directly addressed with sufficient funding and execution, making it a more tractable domain for immediate impact.
The most critical period for a new organization is the year surrounding its founding. Support should be concentrated here, providing grants, networking, and strategic guidance during the pre-launch ideation phase and the immediate post-launch execution phase when it is most vital.
When transitioning careers, it's vital to network and learn aggressively ('hustle'). However, one should resist the urge to accept the first good option ('rush'). The best role is often orders of magnitude better than a merely good one, justifying a patient search.