The danger of AI creating harmful proteins is not in the digital design but in its physical creation. A protein sequence on a computer is harmless. The critical control point is the gene synthesis process. Therefore, biosecurity efforts should focus on providing advanced screening tools to synthesis providers.
Related Insights
The rapid evolution of AI makes reactive security obsolete. The new approach involves testing models in high-fidelity simulated environments to observe emergent behaviors from the outside. This allows mapping attack surfaces even without fully understanding the model's internal mechanics.
The current industry approach to AI safety, which focuses on censoring a model's "latent space," is flawed and ineffective. True safety work should reorient around preventing real-world, "meatspace" harm (e.g., data breaches). Security vulnerabilities should be fixed at the system level, not by trying to "lobotomize" the model itself.
Models designed to predict and screen out compounds toxic to human cells have a serious dual-use problem. A malicious actor could repurpose the exact same technology to search for or design novel, highly toxic molecules for which no countermeasures exist, a risk the researchers initially overlooked.
The field of AI safety is described as "the business of black swan hunting." The most significant real-world risks that have emerged, such as AI-induced psychosis and obsessive user behavior, were largely unforeseen just years ago, while widely predicted sci-fi threats like bioweapons have not materialized.
Contrary to the popular belief that generative AI is easily jailbroken, modern models now use multi-step reasoning chains. They unpack prompts, hydrate them with context before generation, and run checks after generation. This makes it significantly harder for users to accidentally or intentionally create harmful or brand-violating content.
Current biosecurity screens for threats by matching DNA sequences to known pathogens. However, AI can design novel proteins that perform a harmful function without any sequence similarity to existing threats. This necessitates new security tools that can predict a protein's function, a concept termed "defensive acceleration."
Instead of relying on flawed AI guardrails, focus on traditional security practices. This includes strict permissioning (ensuring an AI agent can't do more than necessary) and containerizing processes (like running AI-generated code in a sandbox) to limit potential damage from a compromised AI.
Other scientific fields operate under a "precautionary principle," avoiding experiments with even a small chance of catastrophic outcomes (e.g., creating dangerous new lifeforms). The AI industry, however, proceeds with what Bengio calls "crazy risks," ignoring this fundamental safety doctrine.
The bottleneck for AI in drug development isn't the sophistication of the models but the absence of large-scale, high-quality biological data sets. Without comprehensive data on how drugs interact within complex human systems, even the best AI models cannot make accurate predictions.
Valthos CEO Kathleen, a biodefense expert, warns that AI's primary threat in biology is asymmetry. It drastically reduces the cost and expertise required to engineer a pathogen. The primary concern is no longer just sophisticated state-sponsored programs but small groups of graduate students with lab access, massively expanding the threat landscape.