Unlike a human expert, an LLM's probability estimates and conclusions can be drastically altered by simple rephrasing or irrelevant suggestions. This instability shows they are too easily "pushed around" and lack the coherent world model necessary for trustworthy, high-stakes decision support.

You can't just deploy a probabilistic model like an LLM in a high-stakes field like healthcare. The key is to build a deterministic infrastructure (e.g., a rules engine with clinical guidelines) that governs the AI's operation, ensuring it operates safely within predefined constraints.

Generative AI is designed for creative generation, not consistent output. This core feature makes it unreliable for critical, live applications without human oversight. Humans require predictable patterns, which current AI alone cannot guarantee, making a human at the helm essential for safety and trust.

To maintain trust, AI in medical communications must be subordinate to human judgment. The ultimate guardrail is remembering that healthcare decisions are made by people, for people. AI should assist, not replace, the human communicator to prevent algorithmic control over healthcare choices.

To ensure reliability in healthcare, ZocDoc doesn't give LLMs free rein. It wraps them in a hybrid system where traditional, deterministic code orchestrates the AI's tasks, sets firm boundaries, and knows when to hand off to a human, preventing the 'praying for the best' approach common with direct LLM use.

To overcome resistance, AI in healthcare must be positioned as a tool that enhances, not replaces, the physician. The system provides a data-driven playbook of treatment options, but the final, nuanced decision rightfully remains with the doctor, fostering trust and adoption.

Despite hype in areas like self-driving cars and medical diagnosis, AI has not replaced expert human judgment. Its most successful application is as a powerful assistant that augments human experts, who still make the final, critical decisions. This is a key distinction for scoping AI products.

LLMs are technically non-deterministic systems designed to guess the next most probable word, not verify facts like a calculator. This inherent design means they will confidently produce incorrect information, making human verification indispensable for high-stakes business decisions.

A key risk for AI in healthcare is its tendency to present information with unwarranted certainty, like an "overconfident intern who doesn't know what they don't know." To be safe, these systems must display "calibrated uncertainty," show their sources, and have clear accountability frameworks for when they are inevitably wrong.