An AI model can meet all technical criteria (correctness, relevance) yet produce outputs that are tonally inappropriate or off-brand. Ex-Alexa PM Polly Allen shared how a factually correct answer about COVID was insensitive, proving product leaders must inject human judgment into AI evaluation.

Despite being the focus of the review and positioned as a near-Opus level model, Sonnet 5 performed poorly in the host's final, human-weighted evaluation. The episode, intended to showcase the new model, ironically concluded with it at the bottom of the personal preference leaderboard, behind older models.

When using LLMs to judge other models' output, they consistently rate towards the middle of the curve, akin to humans giving a generic "7 out of 10." These AI judges are not "spiky" enough, failing to recognize unique or exceptional qualities that a human evaluator with strong taste would identify.

A "vibe check" is simply using your brain as a scoring function to intuit if an AI output is good. This aligns with the "do things that don't scale" startup principle and is a necessary first step before building more robust, scalable evaluation systems.

There's a critical paradox in AI evaluation: human experts often agree with the high-level principles and rules given to an AI judge but frequently disagree with the actual judgments it produces. This gap between instruction and application undermines the reliability of AI-driven benchmarking systems.

While AI labs tout performance on standardized tests like math olympiads, these metrics often don't correlate with real-world usefulness or qualitative user experience. Users may prefer a model like Anthropic's Claude for its conversational style, a factor not measured by benchmarks.

Do not blindly trust an LLM's evaluation scores. The biggest mistake is showing stakeholders metrics that don't match their perception of product quality. To build trust, first hand-label a sample of data with binary outcomes (good/bad), then compare the LLM judge's scores against these human labels to ensure agreement before deploying the eval.

A one-size-fits-all evaluation method is inefficient. Use simple code for deterministic checks like word count. Leverage an LLM-as-a-judge for subjective qualities like tone. Reserve costly human evaluation for ambiguous cases flagged by the LLM or for validating new features.

Despite public focus on benchmarks, the market for AI evaluation is profoundly underdeveloped, lacking mature tools, methods, model access, and legal protections. For most non-tech companies, standard benchmarks are irrelevant, forcing reliance on subjective, context-specific, 'vibes-based' assessments.