Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Adit says generic benchmarks are increasingly promotional, so 'state of the art' tells you little about your own data. He recommends testing on a representative distribution and using a thoughtful LLM-as-judge rubric. He notes one public benchmark's published leader got much worse than rivals once documents were tilted or blurred.

Related Insights

Most AI labs evaluate models using public benchmarks, which allows them to effectively train on the test data and inflate scores. Companies like Vals AI maintain private, held-out test sets to provide an uncontaminated, higher-signal measure of a model's real-world intelligence.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Current AI benchmarks have become targets for competition, an example of Goodhart's Law. Models are optimized to top leaderboards rather than develop the general capabilities the benchmarks were designed to measure, creating a false sense of progress and failing to predict real-world performance.

Martone says when a new model appears on Slack or X, he does not trust the public benchmark. His team sets up an internal experiment and compares every model on EliseAI's own benchmarks, because external scores do not reveal performance on real calls at scale.

Companies like Meta are engaging in "chart crimes" to frame new models in the best possible light. By selectively highlighting winning benchmarks (e.g., in blue), they create a visual impression of superiority, even when the model underperforms in other key areas. This signals that benchmarks are becoming marketing tools rather than objective measures.

Model leaderboards are misleading. To ensure a consistent user experience, companies must develop their own evaluation suites reflecting their specific workloads. This allows them to swap underlying models for cost or capability reasons with confidence that the customer-facing outcome remains reliable and high-quality.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

Leading AI companies like OpenAI are publicly discrediting established benchmarks (SuiteBench Pro) and creating their own. This signals a shift where companies use custom benchmarks to highlight their model's strengths, making direct comparisons difficult and forcing users to rely on subjective "vibes" rather than objective standards.

Don't trust academic benchmarks. Labs often "hill climb" or game them for marketing purposes, which doesn't translate to real-world capability. Furthermore, many of these benchmarks contain incorrect answers and messy data, making them an unreliable measure of true AI advancement.

Public AI model benchmarks are becoming a "corporate psyop." The Higgsfield founder claims researchers at large labs are incentivized to game benchmarks for bonuses by contaminating training sets with test data. This artificially inflates scores without improving real-world performance, making benchmarks a poor guide for model selection.