The SWE-bench benchmark is now obsolete primarily because its open-source problems were absorbed into models' training data. This allowed models to 'cheat' by memorizing solutions rather than demonstrating true reasoning, leading to artificially high and meaningless scores.

The most significant gap in AI research is its focus on academic evaluations instead of tasks customers value, like medical diagnosis or legal drafting. The solution is using real-world experts to define benchmarks that measure performance on economically relevant work.

OpenAI's evals team is looking beyond current benchmarks that test self-contained, hour-long tasks. They are calling for new evaluations that measure performance on problems that would take top engineers weeks or months to solve, such as creating entire products end-to-end. This signals a major increase in the complexity and ambition expected from future AI benchmarks.

AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

Don't trust academic benchmarks. Labs often "hill climb" or game them for marketing purposes, which doesn't translate to real-world capability. Furthermore, many of these benchmarks contain incorrect answers and messy data, making them an unreliable measure of true AI advancement.

Despite using nearly 100 software engineers to create 'SWE-Bench Verified', the benchmark had significant flaws, like overly narrow tests that demanded specific, unstated implementation choices. These flaws only became apparent when analyzing why highly capable models were failing, showing that model advancements are necessary to debug and stress-test their own evaluations.

When models achieve suspiciously high scores, it raises questions about benchmark integrity. Intentionally including impossible problems in benchmarks can serve as a flag to test an AI's ability to recognize unsolvable requests and refuse them, a crucial skill for real-world reliability and safety.

A flawed or unsolvable benchmark task can function as a 'canary' or 'honeypot'. If a model successfully completes it, it's a strong signal that the model has memorized the answer from contaminated training data, rather than reasoning its way to a solution.

A major challenge for the 'time horizon' metric is its cost. As AI capabilities improve, the tasks needed to benchmark them grow from hours to weeks or months. The cost of paying human experts for these long durations to establish a baseline becomes extremely high, threatening the long-term viability of this evaluation method.