AI models are surprisingly strong at certain tasks but bafflingly weak at others. This 'jagged frontier' of capability means that experience with AI can be inconsistent. The only way to navigate it is through direct experimentation within one's own domain of expertise.

The future of AI is hard to predict because increasing a model's scale often produces 'emergent properties'—new capabilities that were not designed or anticipated. This means even experts are often surprised by what new, larger models can do, making the development path non-linear.

Unlike traditional software where features are explicitly coded, frontier AI systems are trained on vast datasets, leading to emergent abilities. Their internal mechanisms are not directly designed, which is why developers struggle to reliably instill intended goals and prevent unwanted behaviors.

AI development is more like farming than engineering. Companies create conditions for models to learn but don't directly code their behaviors. This leads to a lack of deep understanding and results in emergent, unpredictable actions that were never explicitly programmed.

We don't fully understand how advanced AI models work. Creators don't program them with explicit knowledge but train them on vast datasets and then run experiments to discover their capabilities. This makes AI development more of a science—studying an unpredictable artifact—than traditional engineering, highlighting an inherent lack of control.

With past shifts like the internet or mobile, we understood the physical constraints (e.g., modem speeds, battery life). With generative AI, we lack a theoretical understanding of its scaling potential, making it impossible to forecast its ultimate capabilities beyond "vibes-based" guesses from experts.

Third-party tracker METR observed that model complexity was doubling every seven months. However, a recent proprietary model shattered this trend, demonstrating nearly double the expected capability for independent operation (15 hours vs. an expected 8). This signals that AI advancement is accelerating unpredictably, outpacing prior scaling laws.

Building machines that learn from vast datasets leads to unpredictable outcomes. OpenAI's GPT-3, trained on text, spontaneously learned to write computer programs—a skill its designers did not explicitly teach it or expect it to acquire. This highlights the emergent and mysterious nature of modern AI.

A profound challenge in AI is that we lack the time to fully evaluate a model's intelligence on long-running tasks. Before we can discover a model's true capabilities, a new, more powerful generation is released, making the previous one obsolete and its full potential unknown.