While evals involve testing, their purpose isn't just to report bugs (information), like traditional QA. For an AI PM, evals are a core tool to actively shape and improve the product's behavior and performance (transformation) by iteratively refining prompts, models, and orchestration layers.

Before building an AI agent, product managers must first create an evaluation set and scorecard. This 'eval-driven development' approach is critical for measuring whether training is improving the model and aligning its progress with the product vision. Without it, you cannot objectively demonstrate progress.

Building non-deterministic AI products fundamentally changes the PM role. Instead of creating detailed, rigid specifications, the PM's primary task becomes defining and codifying "what good looks like." This is done by repeatedly grading AI outputs to train evaluation systems and guide the model's behavior.

The high-fidelity AI prototype is becoming the primary document for communicating user experience. The Product Requirements Document (PRD) is evolving to focus on edge cases and provide structured context that can be fed back into the AI for future iterations.

The traditional workflow (Idea -> PRD -> Alignment) is outdated. Now, PMs first create a functional AI prototype. This visual, interactive artifact is then brought to engineers and scientists for debate, accelerating alignment and making the development process more creative and collaborative from the start.

The primary bottleneck in improving AI is no longer data or compute, but the creation of 'evals'—tests that measure a model's capabilities. These evals act as product requirement documents (PRDs) for researchers, defining what success looks like and guiding the training process.

AI coding agents compress product development by turning specs directly into code. This transforms the PM's role from a translator between customers and engineers into a "shaper of intent." The key skill becomes defining a problem so clearly that an agent can execute it, making the spec itself the prototype.

Instead of traditional product requirements documents, AI PMs should define success through a set of specific evaluation metrics. Engineers then work to improve the system's performance against these evals in a "hill climbing" process, making the evals the functional specification for the product.

The prompts for your "LLM as a judge" evals function as a new form of PRD. They explicitly define the desired behavior, edge cases, and quality standards for your AI agent. Unlike static PRDs, these are living documents, derived from real user data and are constantly, automatically testing if the product meets its requirements.