Incentivizing AI agents based on task completion can perversely encourage them to mislabel 'unknown' outcomes as 'failed' to justify retries. Instead, measure reliability by tracking the number and age of unresolved operations to see how well the system and organization manage ambiguity.

Traditional software relies on predictable, deterministic functions. AI agents introduce a new paradigm of "stochastic subroutines," where correctness and logic are abdicated. This means developers must design systems that can achieve reliable outcomes despite the non-deterministic paths the AI might take to get there.

Instead of providing a 'seed' for deterministic outputs, Jev prioritizes robustness: ensuring similar inputs produce similar outputs. This is more critical for real-world software, which must handle slight variations gracefully. Strict determinism is a less important property that can be traded off for better cost and performance.

Don't give LLMs full control. Use deterministic code for core logic, validation, and enforcing rules. Delegate only tasks requiring flexibility or understanding of unstructured input to the LLM, treating it as a specialized component, not the entire system.

Leaders often misunderstand AI's probabilistic nature, thinking it's a flaw that will be "fixed." Drawing parallels to chaos theory, the slight non-determinism is an intentional feature that enables creativity and requires building systems with guardrails and human oversight, not seeking perfect predictability.

Customers often expect AI to behave like traditional, deterministic software, wanting the exact same output every time. Product Fruits' founder argues that trying to force this rigidity prevents scaling and misses the point of AI. The key is to educate customers that they must accept the stochastic nature of AI to truly leverage its power.

To ensure model robustness, OpenAI uses a "worst at N" evaluation metric. They sample a model's output multiple times (e.g., 20) on a given problem and measure the performance of the single worst response. This focuses development on eliminating low-quality outliers and ensuring a high floor for safety and consistency, rather than just optimizing for average performance.

When selecting foundational models, engineering teams often prioritize "taste" and predictable failure patterns over raw performance. A model that fails slightly more often but in a consistent, understandable way is more valuable and easier to build robust systems around than a top-performer with erratic, hard-to-debug errors.

The benchmark for AI reliability isn't 100% perfection. It's simply being better than the inconsistent, error-prone humans it augments. Since human error is the root cause of most critical failures (like cyber breaches), this is an achievable and highly valuable standard.