[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

The next major advance for AI in software development is not just completing tasks, but deeply understanding entire codebases. This capability aims to "mind meld" the human with the AI, enabling them to collaboratively tackle problems that neither could solve alone.

When models achieve suspiciously high scores, it raises questions about benchmark integrity. Intentionally including impossible problems in benchmarks can serve as a flag to test an AI's ability to recognize unsolvable requests and refuse them, a crucial skill for real-world reliability and safety.

Current benchmarks like SWE-bench test isolated, independent tasks. The new Code Clash benchmark aims to evaluate long-horizon development by having AI models compete in a tournament, continuously improving their own codebases in response to competitive pressure from other models.

Early benchmark improvements focused on adding more languages and repositories. Now, the cutting edge involves creating more difficult evaluation splits through sophisticated curation techniques. Researchers must justify why their new benchmark is qualitatively harder, not just broader, than existing ones.

One vision pushes for long-running, autonomous AI agents that complete complex goals with minimal human input. The counter-argument, emphasized by teams like Cognition, is that real-world value comes from fast, interactive back-and-forth between humans and AI, as tasks are often underspecified.