We scan new podcasts and send you the top 5 insights daily.
State-of-the-art flagship models often overcomplicate basic tasks and consume excessive tokens compared to efficient smaller models. Engineers should reserve top-tier models for orchestration and code review while delegating routine knowledge or computation tasks to cheaper alternatives.
With frontier models costing over 100x more than competent alternatives ($56 vs. 50¢ per million tokens), companies are burning cash. An estimated 98% of tasks sent to top-tier models don't require that power, an inefficiency driven by engineers who are disconnected from cost implications.
Don't use your most powerful and expensive AI model for every task. A crucial skill is model triage: using cheaper models for simple, routine tasks like monitoring and scheduling, while saving premium models for complex reasoning, judgment, and creative work.
A cost-saving workflow is emerging where developers use expensive frontier models for high-level "thinking" and planning stages of a complex task. Once the plan is established, the more routine and high-volume execution steps are routed to cheaper, often open-source, models to optimize both performance and cost.
To manage AI costs effectively, companies should avoid simply capping token usage, as this kills innovation. A better strategy is to build intelligent routers that assess a task's complexity and dynamically route it to the most appropriate model—powerful models for hard tasks, cheaper ones for simple tasks.
Counterintuitively, the most advanced models can be cheaper for simple tasks. As a model approaches perfection, it spends fewer tokens on verification steps like running linters, making it more efficient than smaller models that must constantly check their work.
A sophisticated gateway that routes queries to different models based on complexity is key to managing AI costs. Simple tasks go to cheap, open-source models, while difficult ones use the frontier. This "expert pattern" allows token usage to rise while keeping costs flat.
As AI token consumption becomes a major budget item, companies are moving beyond using a single frontier model. Every organization will need a portfolio of models, including cheaper options for less complex tasks, to manage the "madness" of runaway costs.
To control inference costs, companies are implementing model routing systems. They differentiate between expensive tokens from frontier models for complex reasoning and cheaper tokens from fine-tuned open-source models for simpler workflow tasks. This tiered approach optimizes both performance and budget, avoiding "token maxing."
For repeatable, deterministic workflows, companies can achieve better performance and cost-efficiency by fine-tuning smaller, specialized models. Overusing expensive frontier models for routine tasks is a strategic error in managing "token capital" and AI spend.
An optimal AI architecture routes tasks to different models based on complexity and risk. Simple, low-stakes work like data extraction should go to the cheapest models. Ambiguous, high-stakes work like system design warrants expensive frontier models, where preventing one engineering mistake justifies the premium token cost.