Relying on general-purpose AI benchmarks for specialized applications like medicine or finance is a critical mistake. Domain-specific benchmarks (e.g., MedQA, Finca) are essential to uncover failures that generic tests would miss, preventing potentially disastrous real-world outcomes.

The benchmark for AI performance shouldn't be perfection, but the existing human alternative. In many contexts, like medical reporting or driving, imperfect AI can still be vastly superior to error-prone humans. The choice is often between a flawed AI and an even more flawed human system, or no system at all.

The most significant gap in AI research is its focus on academic evaluations instead of tasks customers value, like medical diagnosis or legal drafting. The solution is using real-world experts to define benchmarks that measure performance on economically relevant work.

Just as standardized tests fail to capture a student's full potential, AI benchmarks often don't reflect real-world performance. The true value comes from the 'last mile' ingenuity of productization and workflow integration, not just raw model scores, which can be misleading.

The concept of a 'correct' clinical output is ambiguous. It requires resolving contradictory chart data, capturing a physician's unstated decision-making, and navigating areas like billing codes where two human experts often disagree. This is a reasoning problem, not just a data problem.

To gain physician trust, AI companies must move beyond proving their algorithm is accurate. The gold standard is large-scale clinical evidence demonstrating tangible improvements in patient outcomes, treatment rates, and decision-making speed.

In a sign of recursive capability improvement, OpenAI found that its model-based grader for the HealthBench evaluation benchmark was more accurate and consistent than the average human physician performing the same grading task. This demonstrates that models can not only perform a task but also evaluate that performance at a superhuman level, a key component of scalable oversight.

The goal isn't for AI to replicate a doctor's thought process, which is constrained by human limitations. Instead, AI should leverage its superior data processing to be fundamentally "truth-seeking," even if it means a longer path to regulatory approval for tasks like prescribing medications.

The most valuable evals aren't built with complex software but are often simple spreadsheets. Their power comes from deep subject matter expertise, which is necessary to create nuanced prompts and accurate scoring criteria that truly test a model's ability in a specific domain like clinical genomics or law.