The most valuable intellectual property for companies will be their unique, private evaluation benchmarks. These evals allow them to "hill climb" any model, ensuring they retain control and are not locked into a single AI provider. The ability to switch models and improve performance is the key asset.

Public internet data has been largely exhausted for training AI models. The real competitive advantage and source for next-generation, specialized AI will be the vast, untapped reservoirs of proprietary data locked inside corporations, like R&D data from pharmaceutical or semiconductor companies.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Frontier AI models exhibit 'jagged intelligence,' excelling at complex tasks like PhD-level science but failing at simple ones like reading a clock. This inconsistency means businesses cannot trust external benchmarks and must create their own internal evaluations based on specific company workflows.

The gap between benchmark scores and real-world performance suggests labs achieve high scores by distilling superior models or training for specific evals. This makes benchmarks a poor proxy for genuine capability, a skepticism that should be applied to all new model releases.

Satya Nadella argues that the most valuable, defensible asset for companies in the AI era will be their proprietary evaluation frameworks. These internal benchmarks allow them to fine-tune any model for their specific needs, ensuring they retain control and avoid vendor lock-in.

The rapid improvement of AI models is maxing out industry-standard benchmarks for tasks like software engineering. To truly understand AI's impact and capability, companies must develop their own evaluation systems tailored to their specific workflows, rather than waiting for external studies.

While general models are powerful, true competitive advantage will come from hyper-specialized AI. This requires training models on vast amounts of proprietary data stored within a company or on a factory floor, creating a moat that general models cannot replicate.

Standardized AI benchmarks are saturated and becoming less relevant for real-world use cases. The true measure of a model's improvement is now found in custom, internal evaluations (evals) created by application-layer companies. Progress for a legal AI tool, for example, is a more meaningful indicator than a generic test score.