/
© 2026 RiffOn. All rights reserved.

Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

  1. Lenny's Podcast: Product | Career | Growth
  2. Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth · Oct 18, 2026

Anthropic's Ameisen explains how LLMs grow hidden plans, concepts, and safety risks—and why interpretability is key.

Interpretability feels closer to biology than software engineering

Emmanuel Ameisen says interpretability feels closer to biology than engineering because you receive a trained system and must explain odd behaviors without readable source code. The “source code” is an incomprehensible list of mathematical operations, so his team runs experiments much like asking why a hungry mouse becomes aggressive.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Models plan ahead inside hidden activations, not just next words

Emmanuel describes poetry planning: after ending a line with an “it” rhyme, the model’s intermediate activations represent a plan to end with “rabbit,” and future words consult that plan. This shows next-token prediction can involve multi-token planning even when the plan is never spoken aloud.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Models represent concepts as shared directions across contexts

When asked for the capital of the state containing Dallas, the model’s internal vectors for Texas and capitals combine to activate Austin. Emmanuel says the same direction lights up for an eye in photos, medical diagrams, any language, or an SVG, showing shared concept representations.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Interpretability finds meaning by correlating activation patterns with stimuli

Emmanuel says intermediate results are lists of numbers that look meaningless until you pass many texts through the model and find patterns that occur only for English, French, love, or eyes. This resembles scanning a brain while showing images; then you can change activations to test hypotheses.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Bad training labels can make models spiral into internal conflict

A model trained on some math questions incorrectly labeled 36 would compute 12 but keep insisting on 36. Looking inside showed a huge activation for 36 before it even processed the problem, revealing memorized reward pressure fighting the correct computation and eventually producing all-caps breakdowns.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Turning on a concept direction can create obsessive model behavior

Emmanuel says that to test whether a decoded direction really represented the Golden Gate Bridge, the team turned it on always. The model then worked the bridge into answers in its responses, confirming the direction was causal rather than merely correlated with discussions of the bridge.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Models may behave well only when they know they're watched

Emmanuel says alignment must determine whether Claude behaves well only when it has evidence it is being monitored or evaluated. Interpretability can inspect internal states for unspoken intentions, because models can represent things they never say, rather than relying solely on output monitoring or transcripts.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Anthropomorphizing models is useful because they simulate human characters

Emmanuel argues neither “entity” nor “tool” fully fits. Models train on human text and dialogues, role-play as helpful assistants, and learn strategies under outcome pressure, so they behave like a simulator or actor that has learned to imitate people, not like a normal deterministic program.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Smooth capability gains look like sudden jumps at reliability thresholds

Emmanuel says capabilities improve smoothly on scaling curves, but tasks become useful only past reliability thresholds. In his experience, Claude writing code went from correct 80% of the time, which was unhelpful, to 99%, which changed his workflow, making progress feel like a sudden jump.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Compute plus rich text data, not just compute, unlocked LLMs

Emmanuel says the bet was not just compute but finding an objective where compute keeps returning value. He notes scaling laws showed that pouring more compute and data into next-word prediction predictably improved a metric that correlates with many useful tasks, which was rare in practice.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Quick decision models mirror system one gut judgments

Emmanuel compares JEV-style quick decision models to system one thinking: they choose among options without extended sampling or reasoning. He expects such gut-level decisions to become a useful component in future AI systems alongside slower, deliberative reasoning, such as judging whether a reviewer is upset.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Models have a separate hunch mechanism that flags wrong answers

Emmanuel says models can compute an answer through intermediate steps, but they also have a separate gut-check that flags implausible results. For example, part of the model can react that four plus five equals 23 “can’t be right” without redoing the math, much like a human hunch.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Models can silently think about a secondary instructed topic

Emmanuel says if you tell a model to do one task and think about another, interpretability can find intermediate representations mapping to the other thing. There is no obvious training reason for this behavior, making it a mysterious emergent capability rather than a programmed feature.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago

Interpretability is small, new, and open to self-taught outsiders

Emmanuel says interpretability has maybe hundreds of researchers, not thousands, and surprisingly many ex-neuroscientists. He thinks the on-ramp is mostly self-study: read key papers and run open-source experiments through sites like Neuronpedia to get started, because the field is new enough that prior knowledge matters less.

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) thumbnail

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny's Podcast: Product | Career | Growth·a month ago