The speed of AI development has created a paradoxical situation where the time to release a new model is shorter than the time required to conduct comprehensive, long-running tests on the previous version. This necessitates new evaluation frameworks, like a 'recall program' for API-based models.

AI's predictive power is based on identifying patterns in historical data. While effective when the future resembles the past, this makes it inherently unable to account for new inventions, crises, or paradigm shifts not represented in its training text. It predicts from old maps, not what will come next in a new world.

AI struggles with long-horizon tasks not just due to technical limits, but because we lack good ways to measure performance. Once effective evaluations (evals) for these capabilities exist, researchers can rapidly optimize models against them, accelerating progress significantly.

To build confidence in AI's ability to forecast the future, researchers are training "historical LLMs" on data ending in a specific year, like 1930. They then test the model's ability to predict text from a later period, like 1940. This process of historical validation helps calibrate and improve models predicting our own future.

If all your evals pass, you don't know the current limits of your system. Evals that consistently fail act as a benchmark. When a new foundation model is released, rerunning these tests immediately reveals if it has overcome previous limitations.

To efficiently assess new AI models, develop a personal portfolio of your most critical tasks. This 'reusable evaluation set,' complete with prompts and success criteria, allows you to quickly and consistently benchmark new models against your specific needs, rather than relying on general capabilities.

An analysis of AI model performance shows a 2-2.5x improvement in intelligence scores across all major players within the last year. This rapid advancement is leading to near-perfect scores on existing benchmarks, indicating a need for new, more challenging tests to measure future progress.

Traditional, point-in-time AI benchmarks are useless because the software stack (models, libraries, drivers) updates constantly, with some libraries deploying twice a week. This relentless optimization requires "living" benchmarks that run continuously to remain relevant.

A profound challenge in AI is that we lack the time to fully evaluate a model's intelligence on long-running tasks. Before we can discover a model's true capabilities, a new, more powerful generation is released, making the previous one obsolete and its full potential unknown.