Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

In personalized AI domains like coaching, outputs lack a single correct response. Product teams should evaluate systems by defining rules against known failure modes—such as prescribing dangerous weights or impossible workout schedules—and measuring specific dimensions like conciseness and appropriate data utilization.

Related Insights

Generic evaluation metrics like "helpfulness" or "conciseness" are vague and untrustworthy. A better approach is to first perform manual error analysis to find recurring problems (e.g., "tour scheduling failures"). Then, build specific, targeted evaluations (evals) that directly measure the frequency of these concrete issues, making metrics meaningful.

The non-deterministic nature of agentic AI makes traditional pass/fail testing insufficient. Businesses must adopt a multi-dimensional scorecard for every interaction, evaluating metrics like compliance, factual accuracy, latency, and intent recognition, not just task completion.

Product managers may lack the expertise to create comprehensive evals from scratch. A better approach is to generate initial outputs with a base model, have subject matter experts review them, and use their direct feedback to define what constitutes a failure. It's easier for experts to spot mistakes than to predict them.

Don't aim for a 100% accurate evaluation system. A good system reveals a 'healthy percentage' of incorrect outputs. Getting excited when evals are wrong is key, as each failure is a clear, actionable opportunity to improve your AI agent.

Don't just rely on explicit feedback like thumbs up/down. Soft signals are powerful evaluation inputs. A user repeatedly re-generating an answer, quickly abandoning a session, or escalating to human support are strong indicators that your AI is failing, even if they don't explicitly say so.

Many people struggle to define what 'good' looks like. Building an evaluation (eval) for an AI system requires you to codify your quality standards, forcing a level of clarity and commitment that improves your own process and the AI's output.

For tasks where a simple right/wrong answer doesn't exist, verification is a major challenge. The solution is creating detailed rubrics with thousands of criteria, often developed with AI help. This provides a granular reward signal that allows models to climb the learning curve even in highly subjective domains.

When providing feedback to AI on subjective tasks like writing, avoid vague comments. Instead, give it quantitative scores on specific dimensions you care about (e.g., clarity: 9/10, wit: 5/10). This gives goal-driven AI a much clearer target for improvement.

While useful for catching regressions like a unit test, directly optimizing for an eval benchmark is misleading. Evals are, by definition, a lagging proxy for the real-world user experience. Over-optimizing for a metric can lead to gaming it and degrading the actual product.

The crucial first step in building evaluations is not to start writing them immediately. Instead, begin by manually reviewing data and creating a structured approach to formalize and externalize your subjective taste and judgment. This provides a solid foundation for all subsequent evals.