The constant release of new AI models has led to "model fatigue." The performance benchmarks promoted by CEOs on social media are often worthless because they omit crucial context like cost and latency, making them irrelevant for real-world application decisions.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

Leading AI companies like OpenAI are publicly discrediting established benchmarks (SuiteBench Pro) and creating their own. This signals a shift where companies use custom benchmarks to highlight their model's strengths, making direct comparisons difficult and forcing users to rely on subjective "vibes" rather than objective standards.

Seemingly simple benchmarks yield wildly different results if not run under identical conditions. Third-party evaluators must run tests themselves because labs often use optimized prompts to inflate scores. Even then, challenges like parsing inconsistent answer formats make truly fair comparison a significant technical hurdle.

Don't trust academic benchmarks. Labs often "hill climb" or game them for marketing purposes, which doesn't translate to real-world capability. Furthermore, many of these benchmarks contain incorrect answers and messy data, making them an unreliable measure of true AI advancement.

When Meta released Llama 4, it excelled on public benchmarks where questions were open source. However, on VALS' private, held-out benchmarks, it significantly underperformed, revealing a major disconnect and showing that self-reported, public scores can be misleading indicators of true capability.

AI labs often use different, optimized prompting strategies when reporting performance, making direct comparisons impossible. For example, Google used an unpublished 32-shot chain-of-thought method for Gemini 1.0 to boost its MMLU score. This highlights the need for neutral third-party evaluation.

Benchmarks comparing AI models are highly sensitive and potentially misleading. Simple changes to the prompt, the evaluation harness, or even the order of multiple-choice answers can flip the rankings. This suggests that headline-grabbing claims of one model's superiority over another are often not robust without deep methodological scrutiny.