We scan new podcasts and send you the top 5 insights daily.
At scale Adit says cost and latency matter alongside accuracy, and a frontier model is far too expensive to apply per page. Reducto aims for frontier-level quality on documents at roughly a cent per page, which he says is orders of magnitude more aggressive on price.
With frontier models costing over 100x more than competent alternatives ($56 vs. 50¢ per million tokens), companies are burning cash. An estimated 98% of tasks sent to top-tier models don't require that power, an inefficiency driven by engineers who are disconnected from cost implications.
For most enterprise tasks, massive frontier models are overkill—a "bazooka to kill a fly." Smaller, domain-specific models are often more accurate for targeted use cases, significantly cheaper to run, and more secure. They focus on being the "best-in-class employee" for a specific task, not a generalist.
Adit argues labs are racing toward superintelligence, and a ChatGPT vision encoder has to handle selfies and image filters. Reducto instead narrows to document tasks it must be exceptional at, which he says is a big part of why a small team can stay ahead.
PMs often default to the most powerful, expensive models. However, comprehensive evaluations can prove that a significantly cheaper or smaller model can achieve the desired quality for a specific task, drastically reducing operational costs. The evals provide the confidence to make this trade-off.
Decagon found that fine-tuning smaller, specialized open-source models led to better performance, lower latency, and reduced costs compared to using general-purpose frontier models for specific enterprise workflows. This contradicts the common "smart but expensive vs. dumb but cheap" tradeoff.
Jev processes tasks up to 400x cheaper than LLMs, with costs as low as cents for thousands of complex queries. This economic shift makes it feasible to analyze entire archives (emails, ads) for deep insights, a task previously too expensive or time-consuming.
Parser's AI costs are lower than its server costs. They achieve this by intentionally avoiding the most powerful, expensive LLMs which are often slow and rate-limited. Instead, they find a balance, prioritizing speed and cost-effectiveness to process high volumes affordably.
The most compelling business reason for enterprises to adopt custom fine-tuning is the need for low latency. For real-time applications like voice bots, large frontier models are too slow. This practical constraint forces companies to use smaller, specialized open-source models.
While frontier models like Claude excel at analyzing a few complex documents, they are impractical for processing millions. Smaller, specialized, fine-tuned models offer orders of magnitude better cost and throughput, making them the superior choice for large-scale, repetitive extraction tasks.
A cheap model that fails often becomes expensive due to retries, fallbacks, and human review. The true measure of economic efficiency is the cost to reliably complete a task, not the raw inference cost, which can be a misleading metric at scale.