Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.

Related Insights

Models are like complex spaghetti code. Interpretability tools can "factor" this code—identifying what neurons and circuits do. The much harder, unsolved problem is "refactoring"—using that knowledge to systematically improve the training process and build a cleaner "codebase" from the start.

The ambition to fully reverse-engineer AI models into simple, understandable components is proving unrealistic as their internal workings are messy and complex. Its practical value is less about achieving guarantees and more about coarse-grained analysis, such as identifying when specific high-level capabilities are being used.

Just as biology deciphers the complex systems created by evolution, mechanistic interpretability seeks to understand the "how" inside neural networks. Instead of treating models as black boxes, it examines their internal parameters and activations to reverse-engineer how they work, moving beyond just measuring their external behavior.

Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.

By analyzing a model predicting Alzheimer's, Goodfire discovered it relied on the length of cell-free DNA fragments—a previously overlooked signal. This demonstrates how interpretability can extract new, testable scientific hypotheses from high-performing "black box" models.

Modern AIs are not programmed with explicit instructions but are trained neural nets, much like a biological brain. We cannot simply "read the code" to understand their reasoning. This "interpretability problem" is a core reason why building superintelligence is so dangerous.

For AI systems to be adopted in scientific labs, they must be interpretable. Researchers need to understand the 'why' behind an AI's experimental plan to validate and trust the process, making interpretability a more critical feature than raw predictive power.

Neural networks, like brains, emerge from countless small nudges during training rather than a premeditated architectural design. The field of interpretability, therefore, functions like neuroscience, attempting to reverse-engineer what this 'evolutionary' process has learned.

Achieving explainability in AI for drug development isn't about post-hoc analysis. It requires building models from the ground up using inherently interpretable data like RNA sequencing and mutational profiles. When the inputs are explainable, the model's outputs become explainable by design.

The goal of AI interpretability is to move beyond "trial and error" model training. By understanding a model's internal computations, developers can shift towards "intentional design," enabling them to debug, edit, and shape AI models with the precision of writing traditional software code, removing current guesswork.