Rather than relying on a small group of experts, OpenAI has built a three-tiered system involving over 260 physicians. This includes high-level strategic advisors, a large cohort for data operations like red-teaming and comparison tasks (communicating via Slack), and a core group of close advisors who translate this collective expertise into concrete evals and training data for researchers.

OpenAI's health division serves a dual purpose: delivering societal benefits and providing a real-world, high-stakes environment for AI safety research. Problems like scalable oversight (supervising superhuman AI) move from theoretical exercises to practical necessities when models outperform physicians on narrow tasks, creating concrete feedback loops that accelerate safety progress.

Singhal invokes AlphaGo's move 37 to describe future health insights that initially surprise experts but later look brilliant. He expects models to integrate years of EHR, chat, and claims data to flag risks humans cannot hold in their heads. Panda Health's antibiotic overuse flag is an early example.

An effective AI strategy in healthcare is not limited to consumer-facing assistants. A critical focus is building tools to augment the clinicians themselves. An AI 'assistant' for doctors to surface information and guide decisions scales expertise and improves care quality from the inside out.

In a sign of recursive capability improvement, OpenAI found that its model-based grader for the HealthBench evaluation benchmark was more accurate and consistent than the average human physician performing the same grading task. This demonstrates that models can not only perform a task but also evaluate that performance at a superhuman level, a key component of scalable oversight.

OpenAI is tackling healthcare from three distinct angles: consumers (queries), clinicians (research), and hospital enterprises (integrations). The vision is not just to serve these groups independently but to build a unified platform where synergies emerge, enabling seamless information sharing and solving systemic bottlenecks.

Recognizing that different specialists have unique information needs, Open Evidence develops distinct AI models for each role. For example, a surgeon's query about a procedure can surface peer-reviewed surgical videos—a feature irrelevant to a medical oncologist—thus enhancing the tool's utility through deep specialization.

Frontier AI models excel in medicine less because of their encyclopedic knowledge and more because of their ability to integrate huge amounts of context. They can synthesize a patient's entire medical history with the latest research—a task difficult for any single human. This highlights that the key to unlocking AI's value is feeding it comprehensive data, as context is the primary driver of superhuman performance.

Karan Singhal says OpenAI adds specialized healthcare data to every training phase for ChatGPT, rather than forking a smaller medical-only model. A separate model would fall behind the frontier too quickly, so OpenAI hitches health expertise to scale and general reasoning improvements. That is the best-of-both-worlds approach he describes.