To avoid saturated evaluations that only confirm existing capabilities, Notion's team creates difficult test suites they expect to fail 70% of the time. This "headroom" provides a clear signal to model providers about frontier needs and helps the team anticipate where the technology is heading.

Meter's choice of a 50% success rate for its viral chart isn't arbitrary. It's the point where measurements are most statistically robust and least sensitive to noise or small sample sizes, unlike higher thresholds like 95% which are harder to resolve accurately.

Constructing a robust eval set involves a process akin to binary search. First, establish a performance floor with an easy, canonical task to ensure basic competency. Then, establish a ceiling with a frontier-level hard task. This maps the model's capabilities and helps fill in the gaps with medium-difficulty prompts.

The primary bottleneck in improving AI is no longer data or compute, but the creation of 'evals'—tests that measure a model's capabilities. These evals act as product requirement documents (PRDs) for researchers, defining what success looks like and guiding the training process.

Building a functional AI agent is just the starting point. The real work lies in developing a set of evaluations ("evals") to test if the agent consistently behaves as expected. Without quantifying failures and successes against a standard, you're just guessing, not iteratively improving the agent's performance.

Don't aim for a 100% accurate evaluation system. A good system reveals a 'healthy percentage' of incorrect outputs. Getting excited when evals are wrong is key, as each failure is a clear, actionable opportunity to improve your AI agent.

If all your evals pass, you don't know the current limits of your system. Evals that consistently fail act as a benchmark. When a new foundation model is released, rerunning these tests immediately reveals if it has overcome previous limitations.

You don't need to create an automated "LLM as a judge" for every potential failure. Many issues discovered during error analysis can be fixed with a simple prompt adjustment. Reserve the effort of building robust, automated evals for the 4-7 most persistent and critical failure modes that prompt changes alone cannot solve.

When an AI agent performs poorly, the most effective solution isn't clever prompt engineering. Braintrust's CEO's strategy is to "close the session" and rewrite the evaluation script from scratch. This forces clarity on the definition of success, which is often the root cause of the agent's failure.