An AI model named EVO2 designed novel bacteriophage genomes from scratch. When created in a lab, these viruses were not only viable but also functioned better than the best-known natural phages at killing E. coli, marking a new era in biological engineering.

Simple refusal mechanisms in AI models are easily bypassed by motivated actors. Effective biosecurity requires deeper interventions, such as curating training data to exclude sensitive biological information or implementing strict access controls for the most powerful models, ensuring they aren't publicly available.

Incidents like AI-generated viruses and agent swarms are not just doomsday previews; they are critical catalysts. They force researchers, policymakers, and the public into an active, global conversation about risks, guardrails, and institutional readiness—the necessary steps to responsibly manage powerful AI capabilities.

Contrary to the focus of many safety frameworks, AI's biggest capability boost is not for novices, who remain incompetent, but for 'mid-tier' actors like PhD students. These individuals have foundational knowledge, making them the most dangerous recipients of AI assistance.

Instead of trying to control open-source AI models, which is intractable, the proposed strategy is to control the small, expensive-to-produce functional datasets they train on. This preserves the beneficial open-source ecosystem while preventing the dissemination of dangerous capabilities like viral design.

Research on bio-foundation models like EVO2 and ESM3 shows that strategically excluding key datasets (e.g., sequences of viruses that infect humans) dramatically reduces a model's performance on dangerous tasks, often to random chance, without harming its useful scientific capabilities.

The belief that nature represents the ceiling of pathogen danger is false. Just as humans engineer materials stronger than any found in nature, AI can be used to design viruses that are far more transmissible or lethal than their natural counterparts.

Other scientific fields operate under a "precautionary principle," avoiding experiments with even a small chance of catastrophic outcomes (e.g., creating dangerous new lifeforms). The AI industry, however, proceeds with what Bengio calls "crazy risks," ignoring this fundamental safety doctrine.

Scientists have designed viruses for years, but AI's breakthrough is making the process thousands of times cheaper and faster. This accelerates positive biotech applications like gene therapy but also heightens biosecurity risks by lowering the barrier to entry for creating problematic agents.