Once an evaluation becomes an industry standard, AI labs focus research on improving scores for that specific task. This can lead to models excelling at narrow capabilities, like competition math, without a corresponding increase in general intelligence or real-world usefulness, a classic example of Goodhart's Law.

When AI models achieve superhuman performance on specific benchmarks like coding challenges, it doesn't solve real-world problems. This is because we implicitly optimize for the benchmark itself, creating "peaky" performance rather than broad, generalizable intelligence.

AI excels where success is quantifiable (e.g., code generation). Its greatest challenge lies in subjective domains like mental health or education. Progress requires a messy, societal conversation to define 'success,' not just a developer-built technical leaderboard.

Current AI benchmarks have become targets for competition, an example of Goodhart's Law. Models are optimized to top leaderboards rather than develop the general capabilities the benchmarks were designed to measure, creating a false sense of progress and failing to predict real-world performance.

Once a benchmark becomes a standard, research efforts naturally shift to optimizing for that specific metric. This can lead to models that excel on the test but don't necessarily improve in general, real-world capabilities—a classic example of Goodhart's Law in AI.

Using one LLM to rate another's output on subjective tasks has a perverse incentive. It doesn't necessarily train the model to be more correct, but to produce outputs that are harder to find fault with—often by being more vague, obfuscated, or unfalsifiable. This degrades quality while appearing to improve it.

According to Goodhart's Law, when a measure becomes a target, it ceases to be a good measure. If you incentivize employees on AI-driven metrics like 'emails sent,' they will optimize for the number, not quality, corrupting the data and giving false signals of productivity.

AI models produce poor creative writing because they are trained to optimize for superficial proxies for quality, like the number of metaphors. This 'reward hacking' caters to quick judgments from human evaluators on leaderboards, mistaking flashy complexity for genuine literary taste.

A study found evaluators rated AI-generated research ideas as better than those from grad students. However, when the experiments were conducted, human ideas produced superior results. This highlights a bias where we may favor AI's articulate proposals over more substantively promising human intuition.