Get your free personalized podcast brief
We scan new podcasts and send you the top 5 insights daily.
Large enterprises building AI agents are not using simple stacks. A major bank's agentic architecture involved 55 distinct components, including various LLMs, frameworks, and databases. This complexity is growing rapidly as companies figure out production requirements like observability, security, and guardrails.
Related Insights
The core needs of AI agents—version control, testing, observability—mirror those of human developers. However, the sheer scale and speed of agentic workflows mean existing tools like Kubernetes are insufficient, requiring a fundamental reimagining of the entire infrastructure stack.
Many companies initially build their own AI gateway, viewing it as a simple, thin proxy layer. However, upon moving agents to production, they quickly discover that real-world complexity around governance, observability, and security requires a far more robust, specialized control plane platform.
Enterprises will not adopt multi-agent AI without two non-negotiable conditions. First, effective guardrails must be in place to ensure safety and compliance. Second, systems must be interoperable, as enterprises will inevitably use agents from diverse vendors like Salesforce, Microsoft, and Google, not a single provider.
Building a functional AI agent demo is now straightforward. However, the true challenge lies in the final stage: making it secure, reliable, and scalable for enterprise use. This is the 'last mile' where the majority of projects falter due to unforeseen complexity in security, observability, and reliability.
Despite the public debate over model dominance, large enterprises are not standardizing on a single type of LLM. Instead, they strategically deploy a portfolio of models—including open source, proprietary, small, and large language models—based on the specific requirements of each use case, from cost to performance.
A complete AI agent solution consists of five distinct layers: an Agent Harness (e.g., Cloud Code), a Search Layer (e.g., Perplexity), a Web Data Layer (e.g., FireCrawl), an Ops Brain (e.g., Obsidian), and an Outbound/Audience layer. Focusing only on the model is insufficient for building a robust product.
The durable investment opportunities in agentic AI tooling fall into three categories that will persist across model generations. These are: 1) connecting agents to data for better context, 2) orchestrating and coordinating parallel agents, and 3) providing observability and monitoring to debug inevitable failures.
Drawing a parallel to the microservices boom, enterprises will soon deploy thousands of AI agents, creating immense operational complexity. The most valuable future products will be those that, like Datadog for microservices, provide governance, monitoring, and orchestration for this sprawling agentic workforce.
Both companies are separating the agent's control layer (harness/brain) from the execution environment (compute/hands). This architectural convergence, driven by enterprise needs for security, durability, and scale, shows a maturing standard for building production-grade AI agents.
Salesforce's Chief AI Scientist explains that a true enterprise agent comprises four key parts: Memory (RAG), a Brain (reasoning engine), Actuators (API calls), and an Interface. A simple LLM is insufficient for enterprise tasks; the surrounding infrastructure provides the real functionality.
