Unlike traditional ML pipelines with predefined steps (DAGs), AI agents operate as dynamic, unrolled graphs. Their workflow is a tree-like structure of LLM and tool calls determined at runtime, requiring systems that can handle real-time graph definition rather than static compilation ahead of time.
To run reliably in the cloud, AI agents cannot be simple synchronous API calls. Their long-running, stateful nature requires an asynchronous architecture. This typically involves a message broker and task queue to farm out agentic loops to ephemeral workers, preventing process failures and enabling scalability.
Unlike stateless applications, updating a live AI agent is highly risky. The system's inherent non-determinism and statefulness mean even a minor change, like altering a system prompt, can have unknown, cascading effects on millions of long-running, in-flight agent executions, making developers terrified of pushing updates.
The path to improving production agents isn't manual analysis but automation via other agents. The vision is for every deployed agent to have a "nurse agent" companion. This trainer constantly analyzes production traces, runs experiments by replaying scenarios with different models or tools, and automatically optimizes the primary agent.
ZenML co-founder Hamza Tahir notes that building durable AI agents—managing non-deterministic code safely and reliably—is essentially a reinvention of core MLOps principles. The fundamental software engineering practices for productionalizing complex, non-deterministic systems are cyclical, moving from DevOps to MLOps and now to AgentOps.
An "agent harness" is the software that translates an LLM's token outputs into actions—the body for the brain. Model providers like Anthropic now tightly couple their models to proprietary harnesses (e.g., Opus 4.8 to Claude Code) via reinforcement learning, making the model self-aware of its environment to boost performance.
While replaying agent execution traces to test changes seems useful, it's scientifically complex. Swapping a model midway through a trace creates an invalid experiment, as the new model might never have made the preceding decisions to reach that specific state. This complicates A/B testing and evaluation for agentic systems.
A key industry tension exists between model providers creating closed, high-performance agent ecosystems (model + harness) and open-source harnesses like LangGraph. The latter camp argues for model-agnosticism to avoid vendor lock-in and ensure business continuity if a specific model is banned or deprecated.