The speed of AI development has created a paradoxical situation where the time to release a new model is shorter than the time required to conduct comprehensive, long-running tests on the previous version. This necessitates new evaluation frameworks, like a 'recall program' for API-based models.

OpenAI's evals team is looking beyond current benchmarks that test self-contained, hour-long tasks. They are calling for new evaluations that measure performance on problems that would take top engineers weeks or months to solve, such as creating entire products end-to-end. This signals a major increase in the complexity and ambition expected from future AI benchmarks.

AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

The viral Meter chart showing exponential AI agent improvement is becoming unreliable. Models like Anthropic's Opus 4.6 are 'saturating' the benchmark's task set, meaning the tool used to measure progress can no longer keep up. The dramatic acceleration may be more a sign of the benchmark's limitations than a pure reflection of capability leaps.

The most sophisticated benchmarks, like Arc AGI, are not meant to be a permanent 'final exam' for AI. They are designed as moving targets that are expected to become saturated and obsolete. This forces researchers to constantly focus on the next most important unsolved problem at the AI frontier.

The term 'GPT-5 Class' is misleading because the frontier of AI advances so rapidly. In this 2026 scenario, the original GPT-5 is already deprecated and scores below the median, not because it degraded, but because benchmarks got harder and the entire field advanced. True comparison requires looking at the current top tier, not a stale version number.

Obsessing over linear model benchmarks is becoming obsolete, akin to comparing dial-up speeds. The real value and locus of competition is moving to the "agentic layer." Future performance will be measured by the ability to orchestrate tools, memory, and sub-agents to create complex outcomes, not just generate high-quality token responses.

An analysis of AI model performance shows a 2-2.5x improvement in intelligence scores across all major players within the last year. This rapid advancement is leading to near-perfect scores on existing benchmarks, indicating a need for new, more challenging tests to measure future progress.

A profound challenge in AI is that we lack the time to fully evaluate a model's intelligence on long-running tasks. Before we can discover a model's true capabilities, a new, more powerful generation is released, making the previous one obsolete and its full potential unknown.