When AI experts say a model 'saturates' a benchmark, it means the test is no longer useful for measuring progress because top models all score near-perfectly. It signals that the evaluation itself has become obsolete, highlighting how quickly AI capabilities are outgrowing our methods of measurement.

Unlike mature tech products with annual releases, the AI model landscape is in a constant state of flux. Companies are incentivized to launch new versions immediately to claim the top spot on performance benchmarks, leading to a frenetic and unpredictable release schedule rather than a stable cadence.

The top-performing Large Language Model has changed multiple times in just a few years, from OpenAI's ChatGPT to Google's Gemini to Anthropic's Claude. This rapid evolution indicates that establishing a durable competitive advantage, or moat, in the foundational model space is extremely difficult.

AI model versioning has moved away from representing specific technical changes and is now primarily a marketing signal. The numbers indicate which competitive "class" a model belongs to (e.g., a "five class" model), and companies may skip versions to appear more advanced, similar to how car manufacturers use model years.

As benchmarks become standard, AI labs optimize models to excel at them, leading to score inflation without necessarily improving generalized intelligence. The solution isn't a single perfect test, but continuously creating new evals that measure capabilities relevant to real-world user needs.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

The most sophisticated benchmarks, like Arc AGI, are not meant to be a permanent 'final exam' for AI. They are designed as moving targets that are expected to become saturated and obsolete. This forces researchers to constantly focus on the next most important unsolved problem at the AI frontier.

An analysis of AI model performance shows a 2-2.5x improvement in intelligence scores across all major players within the last year. This rapid advancement is leading to near-perfect scores on existing benchmarks, indicating a need for new, more challenging tests to measure future progress.

Traditional, point-in-time AI benchmarks are useless because the software stack (models, libraries, drivers) updates constantly, with some libraries deploying twice a week. This relentless optimization requires "living" benchmarks that run continuously to remain relevant.