An AI model's response is not a prediction of what a single user might say, but a probabilistic continuation based on the entirety of its training data—a vast corpus of human writing. Its power stems from this massive-scale pattern matching on our collective cultural output, making it an echo of humanity's written history.
AI's predictive power is based on identifying patterns in historical data. While effective when the future resembles the past, this makes it inherently unable to account for new inventions, crises, or paradigm shifts not represented in its training text. It predicts from old maps, not what will come next in a new world.
Large Language Models learn the structure and language of mathematical solutions from vast text data. This allows them to generate convincing explanations and steps, but they don't perform actual calculations. Their "fluency" in math-like text is different from a calculator's logical execution, leading to confident but incorrect answers.