Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Dan Bender cites a statistic that less than 30% of tokens are consumed by frontier models; most requests can go to open-weight or smaller models. He says a key capability, probed in diligence and monitoring, is whether teams consistently choose when a frontier model is worth it.

Related Insights

As enterprise usage ramps up, LLM token bills threaten corporate budgets, creating intense pressure on CIOs and CFOs to reduce costs. Enterprises do not need frontier-grade intelligence for every workflow; near-frontier open-weight models that trail frontier models by 6 to 12 months can handle 90% of reasoning, coding, and agentic tasks at a third of the cost. This dynamic will force enterprises to downgrade non-critical workloads away from expensive closed models.

To manage AI costs effectively, companies should avoid simply capping token usage, as this kills innovation. A better strategy is to build intelligent routers that assess a task's complexity and dynamically route it to the most appropriate model—powerful models for hard tasks, cheaper ones for simple tasks.

The most advanced AI models are not universally superior; their capabilities form a "jagged frontier." This means organizations can often use more economical, locally-run open-weight models for tasks where they are "good enough," reserving expensive frontier models for specialized needs.

Open-weight models have rapidly matched the quality frontier models had six to eight months prior at dramatically lower token costs. Because users are shifting workloads toward cheaper models, closed-source frontier labs are forced to offer open-weight models through their own APIs to prevent router platforms from siphoning demand.

Instead of relying on a single large AI model, companies are adopting "model orchestration" to control costs. This involves using a router to send prompts to the most appropriate model based on the task, often cascading from cheap, small models to more expensive ones only when necessary.

Companies are building intelligent systems that analyze a user's prompt and automatically route it to the most cost-effective model that can handle the task. This avoids using expensive frontier models for simple requests, with some companies like Coinbase successfully keeping costs flat despite exponential usage growth.

A sophisticated gateway that routes queries to different models based on complexity is key to managing AI costs. Simple tasks go to cheap, open-source models, while difficult ones use the frontier. This "expert pattern" allows token usage to rise while keeping costs flat.

The dream of routing a query to the single "best" model for quality is likely an AI-complete problem. In practice, the primary value of model routers is cost optimization: finding the cheapest model on the Pareto frontier that meets a required quality bar, which is a high priority given expensive token costs.

A production AI agent performs tasks of varying difficulty. Forcing all requests through a single, expensive frontier model is inefficient. A better architecture routes tasks to the most appropriate model: small, cheap open models for high-volume, low-difficulty work like retrieval, reserving the costly frontier API only for high-stakes reasoning where it matters.

To control inference costs, companies are implementing model routing systems. They differentiate between expensive tokens from frontier models for complex reasoning and cheaper tokens from fine-tuned open-source models for simpler workflow tasks. This tiered approach optimizes both performance and budget, avoiding "token maxing."