A core debate in AI is whether LLMs, which are text prediction engines, can achieve true intelligence. Critics argue they cannot because they lack a model of the real world. This prevents them from making meaningful, context-aware predictions about future events—a limitation that more data alone may not solve.

Salesforce's AI Chief warns of "jagged intelligence," where LLMs can perform brilliant, complex tasks but fail at simple common-sense ones. This inconsistency is a significant business risk, as a failure in a basic but crucial task (e.g., loan calculation) can have severe consequences.

Despite advancing capabilities, AI models like ChatGPT can exhibit surprising fragility. They can get stuck in nonsensical loops or "spiral out" on straightforward queries, such as questions about Zapier integrations. This unpredictable fallibility demonstrates that model reliability remains a significant challenge, eroding user trust for critical tasks.

Large Language Models struggle with obvious, real-world facts because their training data (text) over-represents uncertain topics open to debate—the 'maybe sphere.' Bedrock, common-sense knowledge is rarely written down, leaving a significant gap in the AI's world model and creating a need for human oversight on obvious matters.

Generative AI is designed for creative generation, not consistent output. This core feature makes it unreliable for critical, live applications without human oversight. Humans require predictable patterns, which current AI alone cannot guarantee, making a human at the helm essential for safety and trust.

LLMs are technically non-deterministic systems designed to guess the next most probable word, not verify facts like a calculator. This inherent design means they will confidently produce incorrect information, making human verification indispensable for high-stakes business decisions.

Many product builders overestimate current AI capabilities. Understanding AI's limitations, like the non-deterministic nature of LLMs, is more critical than knowing its strengths. Overstating AI's capacity is a direct path to product failure and bad investments.

Contrary to popular belief, generative AI like LLMs may not get significantly more accurate. As statistical engines that predict the next most likely word, they lack true reasoning or an understanding of "accuracy." This fundamental limitation means they will always be prone to making unfixable mistakes.

A primary obstacle for enterprise AI is the 'faithfulness gap' in current LLMs. The justifications these models provide for their outputs often fail to align with the true underlying causes. This discrepancy creates a massive governance and trust issue when using AI for critical, high-stakes decisions.