When teams repeatedly debate the same trade-off (e.g., "job seeker vs. recruiter focus"), it's a signal to create a principle. By making a definitive choice and codifying it (e.g., "Always focus on the job seeker"), you eliminate future arguments and empower teams to make faster, consistent decisions.

When using an LLM to evaluate another AI's output, instruct it to return a binary score (e.g., True/False, Pass/Fail) instead of a numbered scale. Binary outputs are easier to align with human preferences and map directly to the binary decisions (e.g., ship or fix) that product teams ultimately make.

Data that measures success, like a grading rubric, is far more valuable for AI training than simple raw output. This 'second kind of data' enables iterative learning by allowing models to attempt a problem, receive a score, and learn from the feedback.

Do not blindly trust an LLM's evaluation scores. The biggest mistake is showing stakeholders metrics that don't match their perception of product quality. To build trust, first hand-label a sample of data with binary outcomes (good/bad), then compare the LLM judge's scores against these human labels to ensure agreement before deploying the eval.

When creating an "LLM as a judge" to automate evaluations, resist the urge to use a 1-5 rating scale. This creates ambiguity (what does a 3.2 vs 3.7 mean?). Instead, force the judge to make a binary "pass" or "fail" decision. It's a more painful but ultimately more tractable and actionable way to measure quality.

A one-size-fits-all evaluation method is inefficient. Use simple code for deterministic checks like word count. Leverage an LLM-as-a-judge for subjective qualities like tone. Reserve costly human evaluation for ambiguous cases flagged by the LLM or for validating new features.

To avoid bias and misalignment, collaboratively create a weighted decision-making rubric with stakeholders *before* evaluating options. This ensures everyone agrees on the evaluation criteria, making the final decision easier to accept and implement.

When using a 1-5 scale for evaluations, managers often default to the safe middle option (e.g., '3'), which provides ambiguous feedback. By removing the middle number, you force a choice between a positive or negative leaning score, leading to more honest, clear, and actionable assessments.

A standardized decision rubric is ineffective if teams interpret its scores differently (e.g., a '5' means $3M to one PM and $500k to another). To prevent this, have product managers meet regularly to align on how they apply the rubric's criteria and scoring.