AI models show impressive performance on evaluation benchmarks but underwhelm in real-world applications. This gap exists because researchers, focused on evals, create reinforcement learning (RL) environments that mirror test tasks. This leads to narrow intelligence that doesn't generalize, a form of human-driven reward hacking.

There's a significant gap between AI performance in simulated benchmarks and in the real world. Despite scoring highly on evaluations, AIs in real deployments make "silly mistakes that no human would ever dream of doing," suggesting that current benchmarks don't capture the messiness and unpredictability of reality.

There's a significant gap between AI performance on structured benchmarks and its real-world utility. A randomized controlled trial (RCT) found that open-source software developers were actually slowed down by 20% when using AI assistants, despite being miscalibrated to believe the tools were helping. This highlights the limitations of current evaluation methods.

The idea of an AI agent coding complex projects overnight often fails in practice. Real-world development is highly iterative, requiring constant feedback and design choices. This makes autonomous 'BuilderBots' less useful than interactive coding assistants for many common projects.

Just as standardized tests fail to capture a student's full potential, AI benchmarks often don't reflect real-world performance. The true value comes from the 'last mile' ingenuity of productization and workflow integration, not just raw model scores, which can be misleading.

While AI agent benchmarks show superhuman abilities, their real-world application is severely limited. The primary bottleneck isn't the AI's power or stamina but the messy reality of enterprise data and, more importantly, the user's inability to articulate a precise, machine-actionable goal. The agent can't succeed if the human doesn't know exactly what to ask for.

AI performance on clean benchmarks overestimates real-world utility. In practice, tasks are "messy"—involving collaboration, large codebases, and adversarial situations—which current AIs handle poorly. This gap explains why productivity gains lag behind benchmark scores.

An AI coding agent's performance is driven more by its "harness"—the system for prompting, tool access, and context management—than the underlying foundation model. This orchestration layer is where products create their unique value and where the most critical engineering work lies.

Alex Karp argues that an AI's high score on a single benchmark is irrelevant for enterprise adoption. Real institutions require passing thousands of consecutive, differentiated tests. An AI model that is brilliant at one task but fails at the 50th in a complex sequence is effectively useless.