Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

LLMs master syntax because they learn from a bounded manifold of existing code, whereas production operations span an essentially unbounded state space driven by unpredictable workloads. In systems where computational irreducibility prevents direct prediction, software will require empirical experiments and simulations rather than pure static generation.

Related Insights

LLMs shine when acting as a 'knowledge extruder'—shaping well-documented, 'in-distribution' concepts into specific code. They fail when the core task is novel problem-solving where deep thinking, not code generation, is the bottleneck. In these cases, the code is the easy part.

Traditional software relies on predictable, deterministic functions. AI agents introduce a new paradigm of "stochastic subroutines," where correctness and logic are abdicated. This means developers must design systems that can achieve reliable outcomes despite the non-deterministic paths the AI might take to get there.

Models like Fable excel on benchmarks like Frontier Code because the underlying open-source repositories are well-tested and structured for external contributions. Most enterprise codebases lack these "deterministic feedback loops," meaning agentic performance in the real world is far worse than benchmarks suggest. The bottleneck isn't the model, it's the codebase's "agent readiness."

Alex Karp differentiates between hard-coded infrastructure and the 'magical' but limited code generated by LLMs. While LLMs excel at creating probabilistic outputs like dashboards, this 'free code' cannot function as a reliable knowledge store for critical enterprise processes. True enterprise solutions require managed, structured code that understands the world, a feat LLMs alone cannot achieve.

When tasked with building an AI 'harness,' models like GPT and Opus may instinctively generate purely deterministic code, resisting the inclusion of an AI agent within the structure. Developers must prompt very specifically about the desired workflow and where non-deterministic AI components should be integrated.

Current AI coding assistants excel at generating syntactically correct code and boilerplate, but they are poor at software architecture. This high-level, creative task remains the core domain of human developers, positioning AI as a tool for implementation, not design.

An 'LLM-first' approach, where the model handles core logic, creates impressive demos but lacks production reliability. A 'code-first' approach, using code for structure and LLMs for specific tasks, is less flashy but proves robust and debuggable in real-world applications.

Rather than celebrating generic wins, high-performing engineering teams actively map the precise edges where LLMs fail. Discerning which tasks are computational versus narrative keeps teams from shoehorning tabular or deterministic logic into textual forms that trigger severe signal loss and error propagation.

LLMs in production don't often crash spectacularly. Instead, they introduce subtle, probabilistic errors—like incorrect enum values or missing fields—that are hard to debug because they lack clear error patterns, unlike deterministic code failures.

Generative models have largely commoditized code writing, rendering raw syntax output trivial for modern engineering teams. The true leverage lies in upstream clarity—defining specifications, evaluating constraints, validating business logic, and anticipating security risks.

AI Solves Code Generation but Stumbles on Heavy-Tailed Operational Realities | RiffOn