AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

As benchmarks become standard, AI labs optimize models to excel at them, leading to score inflation without necessarily improving generalized intelligence. The solution isn't a single perfect test, but continuously creating new evals that measure capabilities relevant to real-world user needs.

The primary bottleneck in improving AI is no longer data or compute, but the creation of 'evals'—tests that measure a model's capabilities. These evals act as product requirement documents (PRDs) for researchers, defining what success looks like and guiding the training process.

Don't treat your test dataset as static. Monitor online eval scores in production. When you see poor performance, filter for those failing examples and add them to your offline dataset. This ensures your testing evolves with real-world usage patterns.

Despite using nearly 100 software engineers to create 'SWE-Bench Verified', the benchmark had significant flaws, like overly narrow tests that demanded specific, unstated implementation choices. These flaws only became apparent when analyzing why highly capable models were failing, showing that model advancements are necessary to debug and stress-test their own evaluations.

When selecting foundational models, engineering teams often prioritize "taste" and predictable failure patterns over raw performance. A model that fails slightly more often but in a consistent, understandable way is more valuable and easier to build robust systems around than a top-performer with erratic, hard-to-debug errors.

A flawed or unsolvable benchmark task can function as a 'canary' or 'honeypot'. If a model successfully completes it, it's a strong signal that the model has memorized the answer from contaminated training data, rather than reasoning its way to a solution.

Fine-tuning an AI model is most effective when you use high-signal data. The best source for this is the set of difficult examples where your system consistently fails. The processes of error analysis and evaluation naturally curate this valuable dataset, making fine-tuning a logical and powerful next step after prompt engineering.

Instead of waiting for external reports, companies should develop their own AI model evaluations. By defining key tasks for specific roles and testing new models against them with standard prompts, businesses can create a relevant, internal benchmark.