To ensure AI reliability, Salesforce builds environments that mimic enterprise CRM workflows, not game worlds. They use synthetic data and introduce corner cases like background noise, accents, or conflicting user requests to find and fix agent failure points before deployment, closing the "reality gap."

If your application isn't live and you lack real user data, you can still perform evals. The best methods are dogfooding and recruiting friends. If that's not possible, use an LLM to simulate user interactions at scale. This generates the necessary traces to begin the crucial error analysis process before launch.

Teams often mistakenly debate between using offline evals or online production monitoring. This is a false choice. Evals are crucial for testing against known failure modes before deployment. Production monitoring is essential for discovering new, unexpected failure patterns from real user interactions. Both are required for a robust feedback loop.

Treating AI evaluation like a final exam is a mistake. For critical enterprise systems, evaluations should be embedded at every step of an agent's workflow (e.g., after planning, before action). This is akin to unit testing in classic software development and is essential for building trustworthy, production-ready agents.

To mitigate risks like AI hallucinations and high operational costs, enterprises should first deploy new AI tools internally to support human agents. This "agent-assist" model allows for monitoring, testing, and refinement in a controlled environment before exposing the technology directly to customers.

An 'AI SRE' will inevitably destroy a production database without the right primitives. The crucial missing piece isn't better AI, but infrastructure that can safely and cheaply clone production environments for the AI to test its changes before applying them.

To make its AI agents robust enough for production, Sierra runs thousands of simulated conversations before every release. These "AI testing AI" scenarios model everything from angry customers to background noise and different languages, allowing flaws to be found internally before customers experience them.

Public benchmarks are no longer sufficient to prove a model's superiority. The most compelling validation comes from independent tests on proprietary, internal data, as demonstrated by Databricks. This method prevents models from simply "teaching to the test" on public datasets, revealing their true generalization capabilities.

For high-stakes decisions like utilization management, validate an AI model by having it run alongside the existing human process. The AI renders a decision in parallel with the medical director, allowing the organization to confirm alignment and build confidence before “shifting left” to autonomous workflows.