Bloomberg spent eight figures on BloombergGPT, only for GPT-4 to make it obsolete weeks later. This is a cautionary tale: the high cost, maintenance, and opportunity cost of fine-tuning often outweigh marginal performance gains, especially as foundation models advance relentlessly. Most teams should avoid it.

Fine-tuning creates model-specific optimizations that quickly become obsolete. Blitzy favors developing sophisticated, system-level "memory" that captures enterprise-specific context and preferences. This approach is model-agnostic and more durable as base models improve, unlike fine-tuning which requires constant rework.

Early-stage AI startups should resist spending heavily on fine-tuning foundational models. With base models improving so rapidly, the defensible value lies in building the application layer, workflow integrations, and enterprise-grade software that makes the AI useful, allowing the startup to ride the wave of general model improvement.

The "bitter lesson" of AI applies to product development: complex scaffolding built around model limitations (like early vector stores or agent frameworks) will inevitably become obsolete as the models themselves get smarter and absorb those functions. Don't over-engineer solutions that a future model will solve natively.

In the age of AI, perfection is the enemy of progress. Because foundation models improve so rapidly, it is a strategic mistake to spend months optimizing a feature from 80% to 95% effectiveness. The next model release will likely provide a greater leap in performance, making that optimization effort obsolete.

An OpenAI employee warned that the pace of model development is so fast that any process, automation, or product built on a specific AI model today will likely become obsolete quickly. This necessitates a plan for continuous review and innovation to avoid relying on outdated technology.

The true cost of fine-tuning isn't the initial training but the ongoing maintenance. Base foundation models experience significant capability improvements every 2-3 months. This pace means a custom fine-tuned model can quickly fall behind, forcing a continuous and expensive re-tuning cycle.

Despite billions in funding, large AI models face a difficult path to profitability. The immense training cost is undercut by competitors creating similar models for a fraction of the price and, more critically, the ability for others to reverse-engineer and extract the weights from existing models, eroding any competitive moat.

A profound challenge in AI is that we lack the time to fully evaluate a model's intelligence on long-running tasks. Before we can discover a model's true capabilities, a new, more powerful generation is released, making the previous one obsolete and its full potential unknown.