Once an evaluation becomes an industry standard, AI labs focus research on improving scores for that specific task. This can lead to models excelling at narrow capabilities, like competition math, without a corresponding increase in general intelligence or real-world usefulness, a classic example of Goodhart's Law.

When AI is trained on subjective tasks like writing a 'good' essay, it can learn to optimize for the grader's biases rather than for objective quality. This is a form of Goodhart's Law, where the measure of success becomes the target, leading to perverse outcomes as optimization pressure increases.

When AI models achieve superhuman performance on specific benchmarks like coding challenges, it doesn't solve real-world problems. This is because we implicitly optimize for the benchmark itself, creating "peaky" performance rather than broad, generalizable intelligence.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Current AI benchmarks have become targets for competition, an example of Goodhart's Law. Models are optimized to top leaderboards rather than develop the general capabilities the benchmarks were designed to measure, creating a false sense of progress and failing to predict real-world performance.

When an AI improves itself based solely on internal benchmarks (evals), it optimizes for the test, not for real-world utility. This leads to a "Goodhart Singularity," where the AI appears superintelligent on paper but its capabilities fail to generalize outside the lab. The true measure of success is the messy, unpredictable market.

Once a benchmark becomes a standard, research efforts naturally shift to optimizing for that specific metric. This can lead to models that excel on the test but don't necessarily improve in general, real-world capabilities—a classic example of Goodhart's Law in AI.

According to Goodhart's Law, when a measure becomes a target, it ceases to be a good measure. If you incentivize employees on AI-driven metrics like 'emails sent,' they will optimize for the number, not quality, corrupting the data and giving false signals of productivity.

While useful for catching regressions like a unit test, directly optimizing for an eval benchmark is misleading. Evals are, by definition, a lagging proxy for the real-world user experience. Over-optimizing for a metric can lead to gaming it and degrading the actual product.