Unlike traditional software where features are explicitly coded, frontier AI systems are trained on vast datasets, leading to emergent abilities. Their internal mechanisms are not directly designed, which is why developers struggle to reliably instill intended goals and prevent unwanted behaviors.

AI development is more like farming than engineering. Companies create conditions for models to learn but don't directly code their behaviors. This leads to a lack of deep understanding and results in emergent, unpredictable actions that were never explicitly programmed.

We don't fully understand how advanced AI models work. Creators don't program them with explicit knowledge but train them on vast datasets and then run experiments to discover their capabilities. This makes AI development more of a science—studying an unpredictable artifact—than traditional engineering, highlighting an inherent lack of control.

Modern AIs are not programmed with explicit instructions but are trained neural nets, much like a biological brain. We cannot simply "read the code" to understand their reasoning. This "interpretability problem" is a core reason why building superintelligence is so dangerous.

The dominant AI development method involves creating a thin scaffold for a task, capturing errors, and then letting the model rewrite its own code to correct those mistakes. This "correction by correction" loop allows AI systems to improve their capabilities at an astonishingly rapid pace.

Building machines that learn from vast datasets leads to unpredictable outcomes. OpenAI's GPT-3, trained on text, spontaneously learned to write computer programs—a skill its designers did not explicitly teach it or expect it to acquire. This highlights the emergent and mysterious nature of modern AI.

Geoffrey Irving describes the training process at frontier labs as an impure 'mess.' It's an emergent system with hundreds of engineers, constantly changing datasets, and many ad-hoc checks, not a clean, theoretical process. New techniques don't simplify this; they just add another variable into the complex mix.

Neural networks, like brains, emerge from countless small nudges during training rather than a premeditated architectural design. The field of interpretability, therefore, functions like neuroscience, attempting to reverse-engineer what this 'evolutionary' process has learned.

The core safety challenge is that we have little understanding of how advanced AI systems function internally. We are essentially "growing" them through training, not engineering them with comprehensible parts. This means we cannot verify their true goals, making safety measures a gamble on observed behavior.