Anthropic's safety report states that its automated evaluations for high-level capabilities have become saturated and are no longer useful. They now rely on subjective internal staff surveys to gauge whether a model has crossed critical safety thresholds.

Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.

The most significant gap in AI research is its focus on academic evaluations instead of tasks customers value, like medical diagnosis or legal drafting. The solution is using real-world experts to define benchmarks that measure performance on economically relevant work.

Frontier AI models exhibit 'jagged intelligence,' excelling at complex tasks like PhD-level science but failing at simple ones like reading a clock. This inconsistency means businesses cannot trust external benchmarks and must create their own internal evaluations based on specific company workflows.

A "vibe check" is simply using your brain as a scoring function to intuit if an AI output is good. This aligns with the "do things that don't scale" startup principle and is a necessary first step before building more robust, scalable evaluation systems.

AI evaluation shouldn't be confined to engineering silos. Subject matter experts (SMEs) and business users hold the critical domain knowledge to assess what's "good." Providing them with GUI-based tools, like an "eval studio," is crucial for continuous improvement and building trustworthy enterprise AI.

While AI labs tout performance on standardized tests like math olympiads, these metrics often don't correlate with real-world usefulness or qualitative user experience. Users may prefer a model like Anthropic's Claude for its conversational style, a factor not measured by benchmarks.

The rapid improvement of AI models is maxing out industry-standard benchmarks for tasks like software engineering. To truly understand AI's impact and capability, companies must develop their own evaluation systems tailored to their specific workflows, rather than waiting for external studies.

Standardized AI benchmarks are saturated and becoming less relevant for real-world use cases. The true measure of a model's improvement is now found in custom, internal evaluations (evals) created by application-layer companies. Progress for a legal AI tool, for example, is a more meaningful indicator than a generic test score.