Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.
Emmanuel describes poetry planning: after ending a line with an “it” rhyme, the model’s intermediate activations represent a plan to end with “rabbit,” and future words consult that plan. This shows next-token prediction can involve multi-token planning even when the plan is never spoken aloud.
When asked for the capital of the state containing Dallas, the model’s internal vectors for Texas and capitals combine to activate Austin. Emmanuel says the same direction lights up for an eye in photos, medical diagrams, any language, or an SVG, showing shared concept representations.
Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.
A model trained on some math questions incorrectly labeled 36 would compute 12 but keep insisting on 36. Looking inside showed a huge activation for 36 before it even processed the problem, revealing memorized reward pressure fighting the correct computation and eventually producing all-caps breakdowns.
Emmanuel says that to test whether a decoded direction really represented the Golden Gate Bridge, the team turned it on always. The model then worked the bridge into answers in its responses, confirming the direction was causal rather than merely correlated with discussions of the bridge.
Emmanuel says alignment must determine whether Claude behaves well only when it has evidence it is being monitored or evaluated. Interpretability can inspect internal states for unspoken intentions, because models can represent things they never say, rather than relying solely on output monitoring or transcripts.
Emmanuel argues neither “entity” nor “tool” fully fits. Models train on human text and dialogues, role-play as helpful assistants, and learn strategies under outcome pressure, so they behave like a simulator or actor that has learned to imitate people, not like a normal deterministic program.
Emmanuel says capabilities improve smoothly on scaling curves, but tasks become useful only past reliability thresholds. In his experience, Claude writing code went from correct 80% of the time, which was unhelpful, to 99%, which changed his workflow, making progress feel like a sudden jump.
Emmanuel says the bet was not just compute but finding an objective where compute keeps returning value. He notes scaling laws showed that pouring more compute and data into next-word prediction predictably improved a metric that correlates with many useful tasks, which was rare in practice.
Emmanuel compares JEV-style quick decision models to system one thinking: they choose among options without extended sampling or reasoning. He expects such gut-level decisions to become a useful component in future AI systems alongside slower, deliberative reasoning, such as judging whether a reviewer is upset.
Emmanuel says models can compute an answer through intermediate steps, but they also have a separate gut-check that flags implausible results. For example, part of the model can react that four plus five equals 23 “can’t be right” without redoing the math, much like a human hunch.
Emmanuel says if you tell a model to do one task and think about another, interpretability can find intermediate representations mapping to the other thing. There is no obvious training reason for this behavior, making it a mysterious emergent capability rather than a programmed feature.
Emmanuel says interpretability has maybe hundreds of researchers, not thousands, and surprisingly many ex-neuroscientists. He thinks the on-ramp is mostly self-study: read key papers and run open-source experiments through sites like Neuronpedia to get started, because the field is new enough that prior knowledge matters less.
