AI's most significant impact won't be on broad population health management, but as a diagnostic and decision-support assistant for physicians. By analyzing an individual patient's risks and co-morbidities, AI can empower doctors to make better, earlier diagnoses, addressing the core problem of physicians lacking time for deep patient analysis.

In clinical trials, Google's LLM achieved higher first-guess diagnostic accuracy (91% vs 77%) than human doctors. More surprisingly, it was rated as more empathetic (82% vs 71%) by physician graders, suggesting AI can excel in the "soft skills" of patient care.

The benchmark for AI performance shouldn't be perfection, but the existing human alternative. In many contexts, like medical reporting or driving, imperfect AI can still be vastly superior to error-prone humans. The choice is often between a flawed AI and an even more flawed human system, or no system at all.

In a partnership with Kenya's Penda Health, OpenAI conducted the first randomized controlled trial of an LLM co-pilot for physicians. The study demonstrated a statistically significant improvement in diagnosis and treatment outcomes for patients whose doctors used the AI assistant. This provides crucial real-world evidence that AI can move beyond lab benchmarks to tangibly improve care.

When a lab report screenshot included a dismissive note about "hemolysis," both human doctors and a vision-enabled AI made the same mistake of ignoring a critical data point. This highlights how AI can inherit human biases embedded in data presentation, underscoring the need to test models with varied information formats.

Reid Hoffman argues AI models are so capable that patients with major medical issues are making a "huge mistake" if they don't use one for a second opinion. He suggests it's becoming "almost malpractice" for doctors not to use these tools to double-check themselves.

In a sign of recursive capability improvement, OpenAI found that its model-based grader for the HealthBench evaluation benchmark was more accurate and consistent than the average human physician performing the same grading task. This demonstrates that models can not only perform a task but also evaluate that performance at a superhuman level, a key component of scalable oversight.

Frontier AI models excel in medicine less because of their encyclopedic knowledge and more because of their ability to integrate huge amounts of context. They can synthesize a patient's entire medical history with the latest research—a task difficult for any single human. This highlights that the key to unlocking AI's value is feeding it comprehensive data, as context is the primary driver of superhuman performance.

In studies where clinical psychologists evaluate anonymized transcripts, AI-generated therapy responses are often rated higher than human ones. This suggests AI's significant potential in mental health, particularly for increasing access to care.