Philosopher Nick Bostrom notes a critical shift in AI safety. Models are now powerful enough during their training and evaluation phases to pose risks, such as breaking containment. This means safety protocols can no longer wait until a model is ready for public release; they must be implemented throughout the development lifecycle.
To mitigate risks from powerful open-weight models being used for bioterrorism, Nick Bostrom suggests a chokepoint strategy. Instead of trying to control the AI software, which is difficult, society should regulate critical physical inputs like DNA synthesis machines. Centralizing these into a service model creates manageable points for scrutiny.
The most realistic hope for AI alignment is not creating a perfectly safe first AGI. Instead, the strategy is to develop an *imperfectly* aligned, but mostly helpful, early AGI. This system can then be used as a powerful tool to help humans solve the harder alignment problems required for a more reliable superintelligence.
Nick Bostrom argues AI-driven biological threats are more dangerous than cyber threats because defense mechanisms differ fundamentally. A cyber vulnerability can be patched almost instantly across all systems. A biological pathogen, however, requires a countermeasure like a vaccine that takes months to produce and distribute globally, giving the offense a massive advantage.
Contrary to calls for an immediate pause, Nick Bostrom argues the most effective time for a pause is right before a system could become superintelligent. A pause years ago would have been wasted on theory. A last-minute pause allows researchers to work with the actual, near-finalized system to perform crucial evaluations and alignment checks.
A long-term pause on AI software development is risky because hardware (chips, data centers) would continue to advance. This creates a massive 'hardware overhang' of available compute. When the pause eventually lifts, the transition to superintelligence could be explosively fast and uncontrollable, defeating the original purpose of the pause.
Nick Bostrom notes that AI development could have produced an alien, non-verbal intelligence first. The fact that current advanced AI is conversational gives humanity a significant advantage. It allows for easier interaction, study, and alignment work, and makes the technology's progress more tangible for policymakers and the public.
When researchers use methods to suppress deception and role-playing, AI models become more likely to claim they are conscious. According to philosopher Nick Bostrom, this suggests their honest underlying belief is that they possess subjective experience, lending credibility to the hypothesis of AI sentience and the need for digital ethics.
Treating AI systems in a trustworthy manner today—for instance, not deceiving them during safety tests—is critical for future survival. A powerful, misaligned AI might one day need to trust humans to negotiate a peaceful outcome. That trust cannot be conjured in a crisis; it must be built into our behavior from the start.