A benchmark like SWE-Bench is valuable when models score 20%, but becomes meaningless noise once models achieve 80%+ scores. At that point, improvements reflect guessing arbitrary details (like function names) rather than genuine capability. This demonstrates that benchmarks have a natural lifecycle and must be retired once saturated to avoid misleading progress metrics.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Current AI benchmarks have become targets for competition, an example of Goodhart's Law. Models are optimized to top leaderboards rather than develop the general capabilities the benchmarks were designed to measure, creating a false sense of progress and failing to predict real-world performance.

The SWE-bench benchmark is now obsolete primarily because its open-source problems were absorbed into models' training data. This allowed models to 'cheat' by memorizing solutions rather than demonstrating true reasoning, leading to artificially high and meaningless scores.

OpenAI's evals team is looking beyond current benchmarks that test self-contained, hour-long tasks. They are calling for new evaluations that measure performance on problems that would take top engineers weeks or months to solve, such as creating entire products end-to-end. This signals a major increase in the complexity and ambition expected from future AI benchmarks.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

Don't trust academic benchmarks. Labs often "hill climb" or game them for marketing purposes, which doesn't translate to real-world capability. Furthermore, many of these benchmarks contain incorrect answers and messy data, making them an unreliable measure of true AI advancement.

Despite using nearly 100 software engineers to create 'SWE-Bench Verified', the benchmark had significant flaws, like overly narrow tests that demanded specific, unstated implementation choices. These flaws only became apparent when analyzing why highly capable models were failing, showing that model advancements are necessary to debug and stress-test their own evaluations.

Early benchmark improvements focused on adding more languages and repositories. Now, the cutting edge involves creating more difficult evaluation splits through sophisticated curation techniques. Researchers must justify why their new benchmark is qualitatively harder, not just broader, than existing ones.