We scan new podcasts and send you the top 5 insights daily.
When tasked with building an AI 'harness,' models like GPT and Opus may instinctively generate purely deterministic code, resisting the inclusion of an AI agent within the structure. Developers must prompt very specifically about the desired workflow and where non-deterministic AI components should be integrated.
Contrary to the vision of free-wheeling autonomous agents, most business automation relies on strict Standard Operating Procedures (SOPs). Products like OpenAI's Agent Builder succeed by providing deterministic, node-based workflows that enforce business logic, which is more valuable than pure autonomy.
Specialized coding models often fail because a developer's workflow isn't just writing code; it's a complex conversation involving brainstorming, compliance, and web research. The best coding assistants are the most generalist models because every complex task has AGI-like qualities.
Traditional software relies on predictable, deterministic functions. AI agents introduce a new paradigm of "stochastic subroutines," where correctness and logic are abdicated. This means developers must design systems that can achieve reliable outcomes despite the non-deterministic paths the AI might take to get there.
Don't give LLMs full control. Use deterministic code for core logic, validation, and enforcing rules. Delegate only tasks requiring flexibility or understanding of unstructured input to the LLM, treating it as a specialized component, not the entire system.
AI development tools can be "resistant," ignoring change requests. A powerful technique is to prompt the AI to consider multiple options and ask for your choice before building. This prevents it from making incorrect unilateral decisions, such as applying a navigation change to the entire site by mistake.
Karpathy found AI coding agents struggle with genuinely novel projects like his NanoChat repository. Their training on common internet patterns causes them to misunderstand custom implementations and try to force standard, but incorrect, solutions. They are good for autocomplete and boilerplate but not for intellectually intense, frontier work.
Purely agentic systems can be unpredictable. A hybrid approach, like OpenAI's Deep Research forcing a clarifying question, inserts a deterministic workflow step (a "speed bump") before unleashing the agent. This mitigates risk, reduces errors, and ensures alignment before costly computation.
An AI coding agent's performance is driven more by its "harness"—the system for prompting, tool access, and context management—than the underlying foundation model. This orchestration layer is where products create their unique value and where the most critical engineering work lies.
A harness isn't necessarily another AI layer. It's often deterministic code that wraps an AI agent to enforce a specific, repeatable workflow. This 'micromanagement' approach ensures consistency and efficiency for specialized tasks, which general-purpose AI tools lack.
Simply adding AI "nodes" to a deterministic workflow builder is a limited view of AI's potential. This approach fails to capture the human judgment and edge cases that define complex processes. A better architecture empowers AI agents to run standard operating procedures from end to end.