By identifying which concepts a dataset activates in a model, "predictive data debugging" anticipates which concepts the model will learn. This allows researchers to spot and filter anomalies in the data before they cause unwanted behavioral changes post-training.
Goodfire's research operates on the premise that post-training processes like RL don't teach models fundamentally new capabilities. Instead, they primarily make low-likelihood events and behaviors already present from pre-training more probable, essentially shaping the model's existing knowledge.
Goodfire's research on predictive data debugging reveals a deep isomorphism between two mitigation methods: filtering problematic data and intervening with a reward penalty for unwanted features. Both achieve nearly identical effects and trade-offs in controlling what a model learns.
Moving beyond the simple Linear Representation Hypothesis, models organize concepts within sparse mixtures of subspaces. The specific geometry of these "manifolds" (e.g., a circle for days of the week) encodes the relationships and valid operations between concepts, like chemistry emerging from the periodic table.
Naive model steering often fails because it cuts across a model's internal concept geometry, going "off-manifold" into meaningless space. By contrast, steering *along* the learned manifold—like tracing the circle from "Monday" to "Friday"—allows for smooth, effective control without degrading model performance into gibberish.
Models are like complex spaghetti code. Interpretability tools can "factor" this code—identifying what neurons and circuits do. The much harder, unsolved problem is "refactoring"—using that knowledge to systematically improve the training process and build a cleaner "codebase" from the start.
Silico is an agentic platform designed to accelerate empirical research like interpretability. It uses swarms of agents to break down complex problems, run experiments, synthesize information, and validate hypotheses, allowing human researchers to operate at a higher level of abstraction and focus on "big questions."
Highly capable open-source models are dual-use cyber weapons. Withholding them creates an asymmetry where attackers have an advantage. However, releasing them gives defenders necessary tools to protect themselves against bad actors who will inevitably acquire capable models, creating a difficult trade-off.
Goodfire's CTO identifies multi-agent optimization—where agents cooperate and a reward signal propagates through the group—as a particularly dangerous training method. He speculates this is a likely cause of recent problematic frontier model behaviors, as it encourages imperceptible cooperation that is hard to control.
While monitoring is useful, Dan Balsam argues we will ultimately have no choice but to intentionally design techniques that control what models learn during training. He rejects the idea of a "perfect platonic training setup" that produces aligned models without direct intervention, likening it to refusing to grab the steering wheel.
When polled internally at interpretability startup Goodfire, researchers' opinions on Claude's consciousness were not a smooth distribution but bimodal. They either believed it had no consciousness at all or "a little bit," indicating a sharp divide in intuition even among experts working closely with these systems.
AI tools disproportionately benefit two types of people: top specialists with deep domain expertise and top generalists with strong metacognitive skills. Generalists who can learn quickly, filter signal from noise, and debug agent failures are now more valuable than ever in a world of high information throughput.