To measure an AI model's economic value, survey domain experts on how they allocate their time across various tasks. This time-allocation data serves as a proxy for the economic weight of each task, against which the model's performance can be scored.
Related Insights
The biggest opportunity for AI isn't just automating existing human work, but tackling the vast number of valuable tasks that were never done because they were economically inviable. AI and agents thrive on low-cost, high-consistency tasks that were too tedious or expensive for humans, creating entirely new value.
Unlike traditional software that optimizes for time-in-app, the most successful AI products will be measured by their ability to save users time. The new benchmark for value will be how much cognitive load or manual work is automated "behind the scenes," fundamentally changing the definition of a successful product.
Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.
The most significant gap in AI research is its focus on academic evaluations instead of tasks customers value, like medical diagnosis or legal drafting. The solution is using real-world experts to define benchmarks that measure performance on economically relevant work.
OpenAI's new GDPVal framework evaluates AI on real-world knowledge work. It found frontier models produce work rated equal to or better than human experts nearly 50% of the time, while being 100 times faster and cheaper. This provides a direct measure of impending economic transformation.
Even for complex, multi-hour tasks requiring millions of tokens, current AI agents are at least an order of magnitude cheaper than paying a human with relevant expertise. This significant cost advantage suggests that economic viability will not be a near-term bottleneck for deploying AI on increasingly sophisticated tasks.
OpenAI's new GDP-val benchmark evaluates models on complex, real-world knowledge work tasks, not abstract IQ tests. This pivot signifies that the true measure of AI progress is now its ability to perform economically valuable human jobs, making performance metrics directly comparable to professional output.
A major challenge for the 'time horizon' metric is its cost. As AI capabilities improve, the tasks needed to benchmark them grow from hours to weeks or months. The cost of paying human experts for these long durations to establish a baseline becomes extremely high, threatening the long-term viability of this evaluation method.
In the age of AI, software is shifting from a tool that assists humans to an agent that completes tasks. The pricing model should reflect this. Instead of a subscription for access (a license), charge for the value created when the AI successfully achieves a business outcome.
The most valuable AI systems are built by people with deep knowledge in a specific field (like pest control or law), not by engineers. This expertise is crucial for identifying the right problems and, more importantly, for creating effective evaluations to ensure the agent performs correctly.
