Traditional software relies on predictable, deterministic functions. AI agents introduce a new paradigm of "stochastic subroutines," where correctness and logic are abdicated. This means developers must design systems that can achieve reliable outcomes despite the non-deterministic paths the AI might take to get there.

Building non-deterministic AI products fundamentally changes the PM role. Instead of creating detailed, rigid specifications, the PM's primary task becomes defining and codifying "what good looks like." This is done by repeatedly grading AI outputs to train evaluation systems and guide the model's behavior.

With AI agents capable of generating code and designs at an unprecedented rate, the new chokepoint in workflows is human review. The primary challenge is no longer production but scaling the evaluation process to ensure AI-generated output aligns with quality standards and company values.

Treating AI evaluation like a final exam is a mistake. For critical enterprise systems, evaluations should be embedded at every step of an agent's workflow (e.g., after planning, before action). This is akin to unit testing in classic software development and is essential for building trustworthy, production-ready agents.

Building reliable AI agents requires a developer mindset shift. The most critical task is not writing the agent's code but creating robust evaluations ('evals') that define and verify the desired business outcome. This makes a test-driven development approach non-negotiable for enterprise AI.

The primary bottleneck in improving AI is no longer data or compute, but the creation of 'evals'—tests that measure a model's capabilities. These evals act as product requirement documents (PRDs) for researchers, defining what success looks like and guiding the training process.

Unlike traditional software development that starts with unit tests for quality assurance, AI product development often begins with 'vibe testing.' Developers test a broad hypothesis to see if the model's output *feels* right, prioritizing creative exploration over rigid, predefined test cases at the outset.

Building a functional AI agent is just the starting point. The real work lies in developing a set of evaluations ("evals") to test if the agent consistently behaves as expected. Without quantifying failures and successes against a standard, you're just guessing, not iteratively improving the agent's performance.

Comparing AI models based on single, identical prompts is a flawed methodology. A true evaluation involves 'driving' the model through multiple iterations of feedback and correction. This reveals its ability to understand and adapt to your specific intent, which is a far more critical measure of its utility than a single probabilistic output.