When building a PII detector for e-commerce giant Rakuten, Goodfire AI had to train on synthetic data due to privacy rules. This forced them to solve the difficult "synthetic to real" transfer problem to ensure performance on actual customer data, a common enterprise hurdle.

To ensure AI reliability, Salesforce builds environments that mimic enterprise CRM workflows, not game worlds. They use synthetic data and introduce corner cases like background noise, accents, or conflicting user requests to find and fix agent failure points before deployment, closing the "reality gap."

If your application isn't live and you lack real user data, you can still perform evals. The best methods are dogfooding and recruiting friends. If that's not possible, use an LLM to simulate user interactions at scale. This generates the necessary traces to begin the crucial error analysis process before launch.

The impulse to "add AI" is common, but workshops exploring it must first ask "where do we have good, clean data?". Without a solid data foundation, AI ideation is futile. The first innovation step might be improving data collection, not implementing machine learning.

Product teams often use placeholder text and duplicate UI components, but users don't provide good feedback on unrealistic designs. A prototype with authentic, varied content—even if the UI is simpler—will elicit far more valuable user feedback because it feels real.

To test complex AI prompts for tasks like customer persona generation without exposing sensitive company data, first ask the AI to create realistic, synthetic data (e.g., fake sales call notes). This allows you to safely develop and refine prompts before applying them to real, proprietary information, overcoming data privacy hurdles in experimentation.

Early demos shouldn't be used to ask, "Did we build the right thing?" Instead, present them to customers to test your core assumptions and ask, "Did we understand your problem correctly?" This reframes feedback, focusing on the root cause before investing heavily in a specific solution.

Instead of using sensitive company information, you can prompt an AI model to create realistic, fake data for your business. This allows you to experiment with powerful data visualization and analysis workflows without any privacy or security risks.

The 85% AI project failure rate isn't a technology problem. It stems from four business and process issues: failing to identify a narrow use case, using data that isn't clean or ready, not defining success and risk, and applying deterministic Agile methods to probabilistic AI development.