OpenAI's evals team is looking beyond current benchmarks that test self-contained, hour-long tasks. They are calling for new evaluations that measure performance on problems that would take top engineers weeks or months to solve, such as creating entire products end-to-end. This signals a major increase in the complexity and ambition expected from future AI benchmarks.

Mustafa Suleiman measures AI's human-level performance by its practical outputs. He cites an AI's ability to create a daily briefing summary that is superior to what his human chief of staff can produce as a concrete example of achieving human-level performance in a specific, valuable task.

Issues like 'saturation' and 'maxing' reveal a fundamental flaw: benchmarks test narrow, siloed abilities ('Task AGI'). They fail to measure an AI's capacity to combine skills to solve multi-step problems, which is the true bottleneck preventing real-world agentic performance and the next frontier of AI.

Just as standardized tests fail to capture a student's full potential, AI benchmarks often don't reflect real-world performance. The true value comes from the 'last mile' ingenuity of productization and workflow integration, not just raw model scores, which can be misleading.

Traditional AI benchmarks are seen as increasingly incremental and less interesting. The new frontier for evaluating a model's true capability lies in applied, complex tasks that mimic real-world interaction, such as building in Minecraft (MC Bench) or managing a simulated business (VendingBench), which are more revealing of raw intelligence.

The disconnect between AI's superhuman benchmark scores and its limited economic impact exists because many benchmarks test esoteric problems. The Arc AGI prize instead focuses on tasks that are easy for humans, testing an AI's ability to learn new concepts from few examples—a better proxy for general, applicable intelligence.

Alex Karp argues that an AI's high score on a single benchmark is irrelevant for enterprise adoption. Real institutions require passing thousands of consecutive, differentiated tests. An AI model that is brilliant at one task but fails at the 50th in a complex sequence is effectively useless.

Obsessing over linear model benchmarks is becoming obsolete, akin to comparing dial-up speeds. The real value and locus of competition is moving to the "agentic layer." Future performance will be measured by the ability to orchestrate tools, memory, and sub-agents to create complex outcomes, not just generate high-quality token responses.

OpenAI's new GDP-val benchmark evaluates models on complex, real-world knowledge work tasks, not abstract IQ tests. This pivot signifies that the true measure of AI progress is now its ability to perform economically valuable human jobs, making performance metrics directly comparable to professional output.