Current AI benchmarks have become targets for competition, an example of Goodhart's Law. Models are optimized to top leaderboards rather than develop the general capabilities the benchmarks were designed to measure, creating a false sense of progress and failing to predict real-world performance.

OpenAI's evals team is looking beyond current benchmarks that test self-contained, hour-long tasks. They are calling for new evaluations that measure performance on problems that would take top engineers weeks or months to solve, such as creating entire products end-to-end. This signals a major increase in the complexity and ambition expected from future AI benchmarks.

AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

As benchmarks become standard, AI labs optimize models to excel at them, leading to score inflation without necessarily improving generalized intelligence. The solution isn't a single perfect test, but continuously creating new evals that measure capabilities relevant to real-world user needs.

The pursuit of AGI may mirror the history of the Turing Test. Once ChatGPT clearly passed the test, the milestone was dismissed as unimportant. Similarly, as AI achieves what we now call AGI, society will likely move the goalposts and decide our original definition was never the true measure of intelligence.

The most sophisticated benchmarks, like Arc AGI, are not meant to be a permanent 'final exam' for AI. They are designed as moving targets that are expected to become saturated and obsolete. This forces researchers to constantly focus on the next most important unsolved problem at the AI frontier.

As AI achieves impressive milestones, like assisting in creating a cancer vaccine, the public conversation immediately discounts the achievement. The goalposts shift from "AI helped solve a problem" to demanding a fully autonomous, one-shot solution. This pattern of escalating expectations obscures the real, incremental progress being made.

The latest Arc AGI benchmark ditches static puzzles for interactive games with no instructions. This forces models to explore, learn rules, and adapt on the fly. It directly measures their ability to acquire new skills efficiently—a closer proxy for general intelligence than testing memorized reasoning patterns.

Traditional AI benchmarks are seen as increasingly incremental and less interesting. The new frontier for evaluating a model's true capability lies in applied, complex tasks that mimic real-world interaction, such as building in Minecraft (MC Bench) or managing a simulated business (VendingBench), which are more revealing of raw intelligence.