The initial goal of precision psychiatry isn't complex machine learning or perfect biomarkers. It's about systematically collecting basic, meaningful data—like cognitive function—that we already know correlates with treatment outcomes. This simple act of consistent measurement provides a powerful foundation for better understanding patients.

Anthropic intentionally avoids using "user minutes" as a core metric. This strategic choice reflects their focus on safety and user well-being, aiming to build a helpful tool rather than an addictive product. By prioritizing value creation over engagement time, they steer clear of the incentive structures that can lead to psychologically harmful AI behaviors.

OpenAI's health division serves a dual purpose: delivering societal benefits and providing a real-world, high-stakes environment for AI safety research. Problems like scalable oversight (supervising superhuman AI) move from theoretical exercises to practical necessities when models outperform physicians on narrow tasks, creating concrete feedback loops that accelerate safety progress.

Singhal says asking five physicians one question can yield eight answers, so OpenAI built a network of hundreds of experts across specialties, languages, and countries. That network shapes evaluations and training data instead of letting one or two clinicians adjudicate hard cases. It is a deliberate aggregation of expert judgment.

Proving the ROI of clinical AI can take years if based solely on patient outcomes. Instead, focus on early, measurable operational wins that are known proxies for better care. Track metrics like increased clinician capacity and higher patient engagement rates to prove the system's value and build momentum.

Instead of a categorical disease model (virus present/absent), mental health should adopt a dimensional approach like internal medicine. Just as blood pressure exists on a spectrum, psychological traits do too. Treatment decisions can be based on evidence-backed cutoffs for risk, eliminating the need for arbitrary diagnostic boxes.

Singhal says users may want comforting answers, but evals track cases where the model must say the situation is unsafe and the user should seek care. OpenAI treats this as a safety prioritization across training and evaluation, even when users resist the message. That remains central as health usage scales.

In a sign of recursive capability improvement, OpenAI found that its model-based grader for the HealthBench evaluation benchmark was more accurate and consistent than the average human physician performing the same grading task. This demonstrates that models can not only perform a task but also evaluate that performance at a superhuman level, a key component of scalable oversight.

To ensure model robustness, OpenAI uses a "worst at N" evaluation metric. They sample a model's output multiple times (e.g., 20) on a given problem and measure the performance of the single worst response. This focuses development on eliminating low-quality outliers and ensuring a high floor for safety and consistency, rather than just optimizing for average performance.