Despite the hype, Datycs' CEO finds that even fine-tuned healthcare LLMs struggle with the real-world complexity and messiness of clinical notes. This reality check highlights the ongoing need for specialized NLP and domain-specific tools to achieve accuracy in healthcare.
Related Insights
Current LLMs are intelligent enough for many tasks but fail because they lack access to complete context—emails, Slack messages, past data. The next step is building products that ingest this real-world context, making it available for the model to act upon.
![[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor thumbnail](https://assets.flightcast.com/V2Uploads/nvaja2542wefzb8rjg5f519m/01K4D8FB4MNA071BM5ZDSMH34N/square.jpg)
When using AI for complex analysis like a medical case, providing a detailed, unabridged history is crucial. The host found that when he summarized his son's case history to start a new chat, the model's performance noticeably worsened because it lacked the fine-grained, day-to-day data points for accurate trend analysis.
Unlike medical fields requiring physical procedures, psychiatry is fundamentally based on language, assessment, and analysis. This makes it uniquely suited for generative AI applications. Companies are now building fully AI-driven telehealth clinics that handle everything from patient evaluation to billing and clinical trial support.
To ensure reliability in healthcare, ZocDoc doesn't give LLMs free rein. It wraps them in a hybrid system where traditional, deterministic code orchestrates the AI's tasks, sets firm boundaries, and knows when to hand off to a human, preventing the 'praying for the best' approach common with direct LLM use.
People overestimate AI's 'out-of-the-box' capability. Successful AI products require extensive work on data pipelines, context tuning, and continuous model training based on output. It's not a plug-and-play solution that magically produces correct responses.
When a lab report screenshot included a dismissive note about "hemolysis," both human doctors and a vision-enabled AI made the same mistake of ignoring a critical data point. This highlights how AI can inherit human biases embedded in data presentation, underscoring the need to test models with varied information formats.
An effective AI strategy in healthcare is not limited to consumer-facing assistants. A critical focus is building tools to augment the clinicians themselves. An AI 'assistant' for doctors to surface information and guide decisions scales expertise and improves care quality from the inside out.
Off-the-shelf AI models can only go so far. The true bottleneck for enterprise adoption is "digitizing judgment"—capturing the unique, context-specific expertise of employees within that company. A document's meaning can change entirely from one company to another, requiring internal labeling.
The widespread use of AI for health queries is set to change doctor visits. Patients will increasingly arrive with AI-generated analyses of their lab results and symptoms, turning appointments into a three-way consultation between the patient, the doctor, and the AI's findings, potentially improving diagnostic efficiency.
Before complex modeling, the main challenge for AI in biomanufacturing is dealing with unstructured data like batch records, investigation reports, and operator notes. The initial critical task for AI is to read, summarize, and connect these sources to identify patterns and root causes, transforming raw information into actionable intelligence.
