Not all uncertainty is equal. An AI must differentiate between inherent world randomness (aleatoric) and its own lack of knowledge (epistemic). This distinction is critical for deciding whether to act based on probabilities or to gather more information to reduce its ignorance.
An AI model that is confidently wrong is more dangerous and less trustworthy than one that is simply incorrect. As adversarial examples show, the ability for an AI to express calibrated confidence is as important as its raw accuracy for building reliable systems.
To make effective decisions with incomplete information, AI systems require a built-in sense of their own uncertainty. This allows them to act cautiously and adapt when facing unpredictable or novel situations, which is a hallmark of true intelligence.
Current AI models forget old information when learning new things, a problem called "catastrophic forgetting." The Bayesian method, which sequentially updates beliefs with new evidence without discarding priors, offers a natural framework for enabling continual, lifelong AI learning.
Instead of a single prediction, advanced weather models generate a range of possible scenarios. This probabilistic approach, which explicitly represents uncertainty by creating an "ensemble" of forecasts, leads to more robust and accurate overall predictions, which is a counterintuitive result.
Researchers understood the principles of building rational AI systems with uncertainty decades ago. Their adoption was stalled not by theoretical weakness but by intractable computational demands. With modern hardware, these once-abandoned ideas are finally becoming feasible.
When an LLM asserts something confidently, it's not performing a calculation of its certainty. It's predicting the next most probable token, which is often confident-sounding text from its training data. This is why its confidence is fragile and easily swayed.
The goal for AI should be to surpass human rationality, not merely replicate it. Just as a calculator is designed to be better at arithmetic, AI should be built to overcome our cognitive biases and be superior at manipulating probabilities to provide real value.