Systematically review production traces ("open coding"), categorize the observed errors ("axial coding"), and then count them. This simple process transforms subjective "vibe checks" and messy logs into a prioritized, data-backed roadmap for improving your AI application, giving PMs a superpower.

When conducting manual "open coding" for AI evals, teams often get bogged down by trying to reach a consensus. Instead, appoint a single person with deep domain expertise (often the product manager) to be the "benevolent dictator," making the final judgment calls on error categorization. This makes the process tractable and fast.

Product leaders must personally engage with AI development. Direct experience reveals unique, non-human failure modes. Unlike a human developer who learns from mistakes, an AI can cheerfully and repeatedly make the same error—a critical insight for managing AI projects and team workflow.

The common mistake in building AI evals is jumping straight to writing automated tests. The correct first step is a manual process called "error analysis" or "open coding," where a product expert reviews real user interaction logs to understand what's actually going wrong. This grounds your entire evaluation process in reality.

Because PMs deeply understand the customer's job, needs, and alternatives, they are the only ones qualified to write the evaluation criteria for what a successful AI output looks like. This critical task goes beyond technical metrics and is core to the PM's role in the AI era.

Developers often test AI systems with well-formed, correctly spelled questions. However, real users submit vague, typo-ridden, and ambiguous prompts. Directly analyzing these raw logs is the most crucial first step to understanding how your product fails in the real world and where to focus quality improvements.

In AI development, trace analysis is a point of tension. Product Managers should become fluent enough to ask intelligent questions and participate in debugging. However, they should avoid owning the process or tooling, respecting it as engineering's domain to maintain a healthy division of labor.

While senior leaders are trained to delegate execution, AI is an exception. Direct, hands-on use is non-negotiable for leadership. It demystifies the technology, reveals its counterintuitive flaws, and builds the empathy required to understand team challenges. Leaders who remain hands-off will be unable to guide strategy effectively.

AI tools like ChatGPT can analyze traces for basic correctness but miss subtle product experience failures. A product manager's contextual knowledge is essential to identify issues like improper formatting for a specific channel (e.g., markdown in SMS) or failures in user experience that an LLM would deem acceptable.