Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Martone says when a new model appears on Slack or X, he does not trust the public benchmark. His team sets up an internal experiment and compares every model on EliseAI's own benchmarks, because external scores do not reveal performance on real calls at scale.

Related Insights

Most AI labs evaluate models using public benchmarks, which allows them to effectively train on the test data and inflate scores. Companies like Vals AI maintain private, held-out test sets to provide an uncontaminated, higher-signal measure of a model's real-world intelligence.

Public benchmarks typically originate from doctoral dissertations designed for narrow academic inquiries rather than practical product use cases. Over-indexing on these external leaderboards risks building models that excel at viral tests while failing ordinary user needs. Labs must craft rigorous internal evaluation suites tailored to specific product workflows.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Public benchmarks are seen as gamed and counterproductive because true intelligence has a 'je ne sais quoi' that leaderboards can't capture. The ultimate test is not a synthetic score but direct evaluation within a specific workflow. Long-term trust is built on reliability in production, not on winning benchmarks.

Mario Martone says a model that looks reliable at 90% in a paper would mean enormous failures across 60 million yearly calls. He notes five or six test conversations can look good, but at millions of interactions different methods diverge sharply, making evaluation at production scale essential.

Model leaderboards are misleading. To ensure a consistent user experience, companies must develop their own evaluation suites reflecting their specific workloads. This allows them to swap underlying models for cost or capability reasons with confidence that the customer-facing outcome remains reliable and high-quality.

Published benchmarks comparing AI models are merely directional signals due to inconsistencies in test versions, tools, and safety settings. They should be used to decide which workloads to prioritize for internal testing, not as a final verdict for procurement. Local acceptance testing on real tasks remains essential.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

Public benchmarks are no longer sufficient to prove a model's superiority. The most compelling validation comes from independent tests on proprietary, internal data, as demonstrated by Databricks. This method prevents models from simply "teaching to the test" on public datasets, revealing their true generalization capabilities.

Instead of waiting for external reports, companies should develop their own AI model evaluations. By defining key tasks for specific roles and testing new models against them with standard prompts, businesses can create a relevant, internal benchmark.

Ignore public model benchmarks; test on your own calls | RiffOn