Instead of traditional product requirements documents, AI PMs should define success through a set of specific evaluation metrics. Engineers then work to improve the system's performance against these evals in a "hill climbing" process, making the evals the functional specification for the product.
PMs often default to the most powerful, expensive models. However, comprehensive evaluations can prove that a significantly cheaper or smaller model can achieve the desired quality for a specific task, drastically reducing operational costs. The evals provide the confidence to make this trade-off.
Metrics like BLEU and ROUGE compare word overlap, not meaning. An LLM's output like "the cat is on the bed" could be semantically opposite to the ground truth "the cat is on the mat" but still score high. This highlights the need for more sophisticated, meaning-aware evaluations like LLM-as-a-judge.
Instead of manually crafting complex evaluation prompts, a more effective workflow is for a human to define the high-level criteria and red flags. Then, feed this guidance into a powerful LLM to generate the final, detailed, and robust prompt for the evaluation system, as AI is often better at prompt construction.
While evals involve testing, their purpose isn't just to report bugs (information), like traditional QA. For an AI PM, evals are a core tool to actively shape and improve the product's behavior and performance (transformation) by iteratively refining prompts, models, and orchestration layers.
A one-size-fits-all evaluation method is inefficient. Use simple code for deterministic checks like word count. Leverage an LLM-as-a-judge for subjective qualities like tone. Reserve costly human evaluation for ambiguous cases flagged by the LLM or for validating new features.
Complex documents like evaluation strategies are rarely read beforehand. To ensure alignment, adopt the Amazon practice of dedicating the first 15-20 minutes of a kickoff meeting to silent, focused reading. This forces engagement and leads to a more informed and productive discussion.
Product managers may lack the expertise to create comprehensive evals from scratch. A better approach is to generate initial outputs with a base model, have subject matter experts review them, and use their direct feedback to define what constitutes a failure. It's easier for experts to spot mistakes than to predict them.
Don't just rely on explicit feedback like thumbs up/down. Soft signals are powerful evaluation inputs. A user repeatedly re-generating an answer, quickly abandoning a session, or escalating to human support are strong indicators that your AI is failing, even if they don't explicitly say so.