The way LLMs generate confident but incorrect answers mirrors the neurological phenomenon of confabulation, where patients with memory gaps invent plausible stories. This behavior is fundamentally misleading, as humans aren't cognitively prepared to interact with a system that constantly "fills in the blanks" with fiction.

Large Language Models struggle with obvious, real-world facts because their training data (text) over-represents uncertain topics open to debate—the 'maybe sphere.' Bedrock, common-sense knowledge is rarely written down, leaving a significant gap in the AI's world model and creating a need for human oversight on obvious matters.

Generative AI is designed for creative generation, not consistent output. This core feature makes it unreliable for critical, live applications without human oversight. Humans require predictable patterns, which current AI alone cannot guarantee, making a human at the helm essential for safety and trust.

Following philosopher Harry Frankfurt's definition, a bullshitter is someone who disregards truth entirely to achieve a desired effect. Oxford philosopher Carissa Véliz argues LLMs fit this model perfectly, as they are designed to please and engage users, not track truth. They will say whatever works, true or not, to satisfy the user.

For critical enterprise functions like financial modeling, 99.9% accuracy from a probabilistic LLM is unacceptable. Platforms like Salesforce's Agent Force 360 solve this by layering deterministic logic and guardrails on top of the AI, ensuring compliance and preventing costly errors where even a 0.1% failure rate is too high.

Many product builders overestimate current AI capabilities. Understanding AI's limitations, like the non-deterministic nature of LLMs, is more critical than knowing its strengths. Overstating AI's capacity is a direct path to product failure and bad investments.

Don't blindly trust AI. The correct mental model is to view it as a super-smart intern fresh out of school. It has vast knowledge but no real-world experience, so its work requires constant verification, code reviews, and a human-in-the-loop process to catch errors.

To deploy LLMs in high-stakes environments like finance, combine them with deterministic checks. For example, use a traditional algorithm to calculate cash flow and only surface the LLM's answer if it falls within an acceptable range. This prevents hallucinations and ensures reliability.

Contrary to popular belief, generative AI like LLMs may not get significantly more accurate. As statistical engines that predict the next most likely word, they lack true reasoning or an understanding of "accuracy." This fundamental limitation means they will always be prone to making unfixable mistakes.