Constructing a robust eval set involves a process akin to binary search. First, establish a performance floor with an easy, canonical task to ensure basic competency. Then, establish a ceiling with a frontier-level hard task. This maps the model's capabilities and helps fill in the gaps with medium-difficulty prompts.
Large companies like Google and Meta must undergo a painful process of reinventing their "classic consumer software building factory" for the AI era. Startups have a key advantage: they can build AI-native processes and cultures from a blank slate, which is often easier than retrofitting a massive organization.
Evals are designed to guide improvement. A set that scores 100% is too easy and provides no optimization path. A 0% score is too hard and offers no signal. A 25-50% success rate creates a "Goldilocks" zone—a challenging but achievable target for engineering teams to iterate against.
When building an AI feature, a PM's first job isn't writing requirements; it's defining success. This is done by creating an eval set that translates abstract user desires (e.g., "beautiful kitchens" on Pinterest) into concrete, scorable examples, forming the product's true specification.
Unlike Google's consensus-driven approach, Meta's culture under Mark Zuckerberg is highly aggressive and founder-led. This empowers teams with resources and conviction but also creates a high-stakes environment where entire teams, like the Llama 4 team, can be abruptly cut for perceived failures.
Traditional PRDs struggle to describe the vast capabilities of GenAI products. Evals serve as a more effective specification, communicating desired product behavior through a set of concrete examples and expected outcomes, rather than abstract descriptions.
Early LLM benchmarks like MMLU tested question-answering, a now-saturated capability. Today, leading labs like Anthropic evaluate models on agentic tasks involving multi-step reasoning and tool use, reflecting the shift in AI applications from search replacement to automated workflows.
The most valuable evals aren't built with complex software but are often simple spreadsheets. Their power comes from deep subject matter expertise, which is necessary to create nuanced prompts and accurate scoring criteria that truly test a model's ability in a specific domain like clinical genomics or law.