We scan new podcasts and send you the top 5 insights daily.
Emmanuel says interpretability has maybe hundreds of researchers, not thousands, and surprisingly many ex-neuroscientists. He thinks the on-ramp is mostly self-study: read key papers and run open-source experiments through sites like Neuronpedia to get started, because the field is new enough that prior knowledge matters less.
The quest to understand AI models is mirroring neuroscience's historical path. Researchers first saw representations as distributed, then focused on single neurons ('neuron doctrine'), and now study how populations of neurons encode complex concepts—replicating neuroscience's shift to the 'population doctrine.'
Anthropic's work on reading a model's internal "thoughts" is more than a safety feature; it's a new frontier for performance. The ability to "train the thoughts, not just the words" gives developers a direct lever to improve a model's internal reasoning, fix failures, and enhance reliability, moving interpretability from theory to practice.
Mechanistic interpretability (Mekinterp) research has been slow due to its manual, ad-hoc nature. The guests argue that coding agents can automate the experimentation process, enabling large-scale, systematic analysis of AI models. The first science AI should automate is the science of understanding itself.
Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.
The field is moving beyond labeling concepts with sparse autoencoders. The new frontier is understanding the intricate geometric structures (manifolds) these concepts form in a model's latent space and how circuits transform them, providing a more unified, dynamic view.
Pessimism around deep neural networks acting as uninterpretable black boxes is shifting as automated scientist agents probe network layers. Modern tooling reveals that trillions of parameter connections are largely legible without needing reliance on sparse autoencoders to untangle overlapping neuron features.
Just as biology deciphers the complex systems created by evolution, mechanistic interpretability seeks to understand the "how" inside neural networks. Instead of treating models as black boxes, it examines their internal parameters and activations to reverse-engineer how they work, moving beyond just measuring their external behavior.
Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.
Access to frontier models is not a prerequisite for impactful AI safety research, particularly in interpretability. Open-source models like Llama or Qwen are now powerful enough ("above the waterline") to enable world-class research, democratizing the field beyond just the major labs.
Neural networks, like brains, emerge from countless small nudges during training rather than a premeditated architectural design. The field of interpretability, therefore, functions like neuroscience, attempting to reverse-engineer what this 'evolutionary' process has learned.