Before going live, top teams run the AI system in parallel with existing workflows, processing real production traffic without exposing the output. This "shadow mode" provides an honest accuracy benchmark on unfiltered data and is treated as a non-negotiable step to de-risk the launch.

If your application isn't live and you lack real user data, you can still perform evals. The best methods are dogfooding and recruiting friends. If that's not possible, use an LLM to simulate user interactions at scale. This generates the necessary traces to begin the crucial error analysis process before launch.

Joon Sung Park notes they are observing the beginnings of a scaling law for simulation. As they ingest more compute and high-quality human data, they see predictable improvements in the model's ability to accurately predict and simulate human behavior.

To convince skeptical stakeholders of AI's value, first validate the model against past surveys to show its responses align with human results most of the time. This baseline of trust makes the small percentage of divergent, interesting signals more credible and actionable, rather than being dismissed as model error.

LLMs trained on online text often reflect what people say, not what they do. Simile bridges this 'say-do gap' by collecting real behavioral data and personal life stories through partners like Gallup. This grounds their agent simulations in reality, making them more predictive of actual behavior.

To ensure product quality, Fixer pitted its AI against 10 of its own human executive assistants on the same tasks. They refused to launch features until the AI could consistently outperform the humans on accuracy, using their service business as a direct training and validation engine.

To make its AI agents robust enough for production, Sierra runs thousands of simulated conversations before every release. These "AI testing AI" scenarios model everything from angry customers to background noise and different languages, allowing flaws to be found internally before customers experience them.

A common misconception is that simulation perfectly represents reality. In practice, it's a continuous loop: real-world data is required to tune simulator parameters, and this validation must be repeated until the gap between simulation and reality is small enough to trust the results.

It's impossible to generate human data at the scale of in silico experiments. The key is to create highly accurate simulations of human physiology (digital twins) and then validate their predictions with limited, strategic human data. If the model proves reliable, it could drastically accelerate R&D.