Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Jassy says inference costs have fallen but remain an order of magnitude too expensive. Enterprises considering rewriting CPU applications for AI accelerators face a high benefit bar when costs stay high. Lower inference costs would make pervasive adoption easier, and Amazon's Trainium chips target better price performance.

Related Insights

Google's rumored "Gemini 3.2 Flash" model suggests a strategy focused on cost-efficiency rather than chasing state-of-the-art benchmarks. By offering near-frontier performance at a 15-20x lower inference cost, Google can capture a huge segment of the enterprise market focused on practical, scalable implementation.

Despite clear ROI, Glean's founder argues current AI costs are "absurdly expensive," citing a single internal engineering triage agent that cost one million dollars per month. He believes this is a historical anomaly and predicts that competition and open source will force inference prices to drop by orders of magnitude.

The dominant use of AI compute is moving from training massive models to running inference tasks. This shift fundamentally alters the market, enabling broader enterprise adoption via cheaper, open models and changing the demand profile for compute hardware beyond the absolute cutting edge.

A primary risk for major AI infrastructure investments is not just competition, but rapidly falling inference costs. As models become efficient enough to run on cheaper hardware, the economic justification for massive, multi-billion dollar investments in complex, high-end GPU clusters could be undermined, stranding capital.

The primary short-term risk for the AI sector isn't capital expenditure but the high cost of token generation. For AI applications to become ubiquitous, the unit economics must improve. If running a single query remains prohibitively expensive for businesses, widespread, sustainable adoption will be impossible, threatening the entire investment thesis.

Jassy describes AI demand as a barbell: AI labs and a few runaway apps consume massive compute, while enterprises today mostly use AI for cost avoidance or productivity. The middle—current production enterprise apps—has barely adopted inference, but he expects it to become the largest absolute segment.

Enterprises currently suffer from a shortage of in-house AI skills and experience, creating a gap between model capabilities and real enterprise deployment. As model inference costs drop and safety mechanisms solidify, Jevons paradox will take effect: reduced unit costs will not reduce total spend, but will instead make enterprises comfortable deploying AI across far more workflows. Consequently, the enterprise market will expand massively rather than become a zero-sum game.

Drawing a parallel to AWS's history, AI inference costs are expected to continuously decrease over time. As usage skyrockets, providers will be incentivized to lower prices to capture market share, making fears of escalating costs for startups unlikely to materialize.

Arvind Krishna forecasts a 1000x drop in AI compute costs over five years. This won't just come from better chips (a 10x gain). It will be compounded by new processor architectures (another 10x) and major software optimizations like model compression and quantization (a final 10x).

The business case for AI is strong, as executing a task for $2-$5 via AI can save an enterprise $55. This significant return on investment suggests companies are financially motivated to increase, not decrease, their spending on AI services over time, despite current market concerns.