The cost of re-validating, QA-ing, and re-training internal apps built on a specific LLM far outweighs potential token savings. Once an application is "dialed in" on a model like Claude Opus, the business has little incentive to switch, creating a durable competitive advantage.

The speed of AI development has created a paradoxical situation where the time to release a new model is shorter than the time required to conduct comprehensive, long-running tests on the previous version. This necessitates new evaluation frameworks, like a 'recall program' for API-based models.

Despite access to state-of-the-art models, most ChatGPT users defaulted to older versions. The cognitive load of using a "model picker" and uncertainty about speed/quality trade-offs were bigger barriers than price. Automating this choice is key to driving mass adoption of advanced AI reasoning.

The true building block of an AI feature is the "agent"—a combination of the model, system prompts, tool descriptions, and feedback loops. Swapping an LLM is not a simple drop-in replacement; it breaks the agent's behavior and requires re-engineering the entire system around it.

Unlike consumer chatbots, organizations like the Pentagon that deeply integrate an AI model's API and tech stack into their operations face significant costs and disruption when trying to switch providers.

Unlike traditional APIs, LLMs are hard to abstract away. Users develop a preference for a specific model's 'personality' and performance (e.g., GPT-4 vs. 3.5), making it difficult for applications to swap out the underlying model without user notice and pushback.

The OpenAI Codex app would have "absolutely failed" if launched three months earlier. The only difference was the underlying model's capability. This reveals a new product risk: a perfectly designed product can fail simply because the AI isn't smart enough yet, requiring teams to relaunch ideas as models improve.

To fully leverage rapidly improving AI models, companies cannot just plug in new APIs. Notion's co-founder reveals they completely rebuild their AI system architecture every six months, designing it around the specific capabilities of the latest models to avoid being stuck with suboptimal implementations.

Despite constant new model releases, enterprises don't frequently switch LLMs. Prompts and workflows become highly optimized for a specific model's behavior, creating significant switching costs. Performance gains of a new model must be substantial to justify this re-engineering effort.