Bloomberg spent eight figures on BloombergGPT, only for GPT-4 to make it obsolete weeks later. This is a cautionary tale: the high cost, maintenance, and opportunity cost of fine-tuning often outweigh marginal performance gains, especially as foundation models advance relentlessly. Most teams should avoid it.

Fine-tuning creates model-specific optimizations that quickly become obsolete. Blitzy favors developing sophisticated, system-level "memory" that captures enterprise-specific context and preferences. This approach is model-agnostic and more durable as base models improve, unlike fine-tuning which requires constant rework.

Early-stage AI startups should resist spending heavily on fine-tuning foundational models. With base models improving so rapidly, the defensible value lies in building the application layer, workflow integrations, and enterprise-grade software that makes the AI useful, allowing the startup to ride the wave of general model improvement.

The "bitter lesson" of AI applies to product development: complex scaffolding built around model limitations (like early vector stores or agent frameworks) will inevitably become obsolete as the models themselves get smarter and absorb those functions. Don't over-engineer solutions that a future model will solve natively.

In the age of AI, perfection is the enemy of progress. Because foundation models improve so rapidly, it is a strategic mistake to spend months optimizing a feature from 80% to 95% effectiveness. The next model release will likely provide a greater leap in performance, making that optimization effort obsolete.

An OpenAI employee warned that the pace of model development is so fast that any process, automation, or product built on a specific AI model today will likely become obsolete quickly. This necessitates a plan for continuous review and innovation to avoid relying on outdated technology.

While early AI development requires constant testing of new models, Conative.ai found they eventually reached a stable architecture. The focus then shifted from wholesale model replacement to fine-tuning existing layers with specific data, reducing the pressure to chase every new innovation.

The AI landscape is uniquely challenging due to the rapid depreciation of both models (new ones top leaderboards weekly) and hardware (Nvidia launched three new SKUs in one year). This creates a constant, complex management burden, justifying the need for platforms that abstract away these choices.

Unlike traditional, long-lasting infrastructure, AI skills have a short half-life due to rapid model updates and changing contexts. Treat them as iterative, ephemeral assets that must be re-evaluated on a monthly basis to remain effective.