We scan new podcasts and send you the top 5 insights daily.
Mike Taylor, Every's head of evals, takes qualitative model impressions and turns them into personal benchmarks and explicit tests, such as defining what "sycophantic" means. The goal is to find capability overhang and test models beyond gut feel, then share the benchmarks as an internal product.
The concept of a single, standardized test for AI is becoming obsolete. The future lies in personalized evaluations tailored to an organization's or individual's specific values and goals. For example, a progressive politician and a conservative one would need different benchmarks to assess if an AI aligns with their objectives.
As models reach peak intelligence on standard benchmarks, qualitative evaluations become critical. The speaker adopts a "psychologist hat," asking models about their self-perception and relationship with the user to reveal deeper insights into their personality, biases, and alignment than traditional tests can provide.
Quantitative benchmarks are insufficient for choosing a daily tool. A qualitative, hands-on "vibe check" across diverse, personalized tasks reveals a model's true utility, personality, and joy of use. This subjective method is crucial for reflecting real-world workflow integration and user satisfaction.
A "vibe check" is simply using your brain as a scoring function to intuit if an AI output is good. This aligns with the "do things that don't scale" startup principle and is a necessary first step before building more robust, scalable evaluation systems.
Standard benchmarks are insufficient. A more effective evaluation method is a hybrid approach, weighting a human's qualitative 'taste test' (e.g., 70%) more heavily than an LLM judge's automated score (e.g., 30%). This prioritizes subjective qualities like design, usability, and writing style.
As AI models saturate traditional benchmarks, their scores become meaningless. The most effective evaluation is now a personal "vibe check" based on the Gell-Mann Amnesia principle: test the model in a domain you know intimately. If it impresses you there, its capabilities are likely real and not just superficially plausible.
To efficiently assess new AI models, develop a personal portfolio of your most critical tasks. This 'reusable evaluation set,' complete with prompts and success criteria, allows you to quickly and consistently benchmark new models against your specific needs, rather than relying on general capabilities.
The host's personal "vibe check" rankings of AI models were the inverse of the scores from an automated, LLM-judged benchmark. This highlights the gap between quantitative metrics and subjective human taste, suggesting that relying solely on AI judges misses crucial aspects of quality and real-world usability.
Traditional AI benchmarks are becoming meaningless as models quickly saturate them. The best way to evaluate a new model is to apply it to a subject you know intimately and see if it triggers the 'Gell-Mann Amnesia' effect. This qualitative, domain-specific 'vibe check' is a more reliable indicator of true capability than abstract scores.
Instead of waiting for external reports, companies should develop their own AI model evaluations. By defining key tasks for specific roles and testing new models against them with standard prompts, businesses can create a relevant, internal benchmark.