Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.

PMs often default to the most powerful, expensive models. However, comprehensive evaluations can prove that a significantly cheaper or smaller model can achieve the desired quality for a specific task, drastically reducing operational costs. The evals provide the confidence to make this trade-off.

Leading AI companies like OpenAI are publicly discrediting established benchmarks (SuiteBench Pro) and creating their own. This signals a shift where companies use custom benchmarks to highlight their model's strengths, making direct comparisons difficult and forcing users to rely on subjective "vibes" rather than objective standards.

Public benchmarks are no longer sufficient to prove a model's superiority. The most compelling validation comes from independent tests on proprietary, internal data, as demonstrated by Databricks. This method prevents models from simply "teaching to the test" on public datasets, revealing their true generalization capabilities.

Traditional, static benchmarks for AI models go stale almost immediately. The superior approach is creating dynamic benchmarks that update constantly based on real-world usage and user preferences, which can then be turned into products themselves, like an auto-routing API.

The rapid improvement of AI models is maxing out industry-standard benchmarks for tasks like software engineering. To truly understand AI's impact and capability, companies must develop their own evaluation systems tailored to their specific workflows, rather than waiting for external studies.

Standardized AI benchmarks are saturated and becoming less relevant for real-world use cases. The true measure of a model's improvement is now found in custom, internal evaluations (evals) created by application-layer companies. Progress for a legal AI tool, for example, is a more meaningful indicator than a generic test score.

The rapid release of new AI models makes it crucial for companies to move beyond industry benchmarks. Developing internal evaluation systems ("evals") is necessary to test and determine which model performs best for unique, high-value business use cases, as model choice is becoming extremely important.