Faced with rising costs from proprietary labs, sophisticated enterprise clients are building internal evaluation and routing systems. This allows them to use cheaper, open-source models for less complex tasks, optimizing for both cost and performance.

While public benchmarks show general model improvement, they are almost orthogonal to enterprise adoption. Enterprises don't care about general capabilities; they need near-perfect precision on highly specific, internal workflows. This requires extensive fine-tuning and validation, not chasing leaderboard scores.

To navigate millions of public models, Dell built a Model Evaluation Protocol (MEP). This system uses AI agents to test new models across different hardware, plotting performance vs. intelligence on a Pareto curve to provide tailored, data-driven customer recommendations.

Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.

Warp replays its own past development tasks using different LLMs to generate internal benchmarks. This data provides concrete evidence for a model routing strategy, allowing them to choose the optimal model based on their specific cost-quality tradeoffs for different types of tasks.

Model leaderboards are misleading. To ensure a consistent user experience, companies must develop their own evaluation suites reflecting their specific workloads. This allows them to swap underlying models for cost or capability reasons with confidence that the customer-facing outcome remains reliable and high-quality.

PMs often default to the most powerful, expensive models. However, comprehensive evaluations can prove that a significantly cheaper or smaller model can achieve the desired quality for a specific task, drastically reducing operational costs. The evals provide the confidence to make this trade-off.

A cost-saving workflow is emerging where developers use expensive frontier models for high-level "thinking" and planning stages of a complex task. Once the plan is established, the more routine and high-volume execution steps are routed to cheaper, often open-source, models to optimize both performance and cost.

The smartest 'AI-pilled' companies adopt a two-tiered model strategy. They use expensive, frontier models for internal, high-leverage tasks like creating new knowledge and optimizing processes. However, they use cheaper, open-weight models in the 'bill of materials' for the customer-facing product to manage costs effectively.