With frontier models costing over 100x more than competent alternatives ($56 vs. 50¢ per million tokens), companies are burning cash. An estimated 98% of tasks sent to top-tier models don't require that power, an inefficiency driven by engineers who are disconnected from cost implications.

Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.

The most significant gap in AI research is its focus on academic evaluations instead of tasks customers value, like medical diagnosis or legal drafting. The solution is using real-world experts to define benchmarks that measure performance on economically relevant work.

Just as standardized tests fail to capture a student's full potential, AI benchmarks often don't reflect real-world performance. The true value comes from the 'last mile' ingenuity of productization and workflow integration, not just raw model scores, which can be misleading.

Traditional AI benchmarks with percentage-based scores often saturate, losing their signal as models improve. Evals like VendingBench, which measure performance in dollars, have no upper ceiling. This provides a more durable and meaningful way to track AI progress and capabilities compared to finite scoring systems.

Traditional AI benchmarks are seen as increasingly incremental and less interesting. The new frontier for evaluating a model's true capability lies in applied, complex tasks that mimic real-world interaction, such as building in Minecraft (MC Bench) or managing a simulated business (VendingBench), which are more revealing of raw intelligence.

Evaluating AI models on cost-per-token is misleading because it ignores the hidden cost of human labor to fix failures. The true 'cost per successful task' is a business metric that accounts for both the API invoice and the payroll expense for rework, revealing a more accurate total cost of ownership.

OpenAI's new GDP-val benchmark evaluates models on complex, real-world knowledge work tasks, not abstract IQ tests. This pivot signifies that the true measure of AI progress is now its ability to perform economically valuable human jobs, making performance metrics directly comparable to professional output.

An AI model might have a low cost per token but be 'token hungry,' requiring more tokens to complete a task. This makes it more expensive overall than a model with a higher per-token cost but greater efficiency. Evaluating models on a 'cost per task' basis provides a more accurate ROI.