Unlike older methods that use an "unknown" placeholder, modern tokenizers built on Byte-Pair Encoding (BPE) can represent any string. When faced with a new word, they don't crash or lose information. Instead, they break it down into the smallest known components, ensuring universal representability.

An LLM's core training objective—predicting the next token—makes it sensitive to the raw frequency of words and numbers online. This creates a subtle but profound flaw: it's more likely to output '30' than '29' in a counting task, not because of logic, but because '30' is statistically more common in its training data.

When an LLM asserts something confidently, it's not performing a calculation of its certainty. It's predicting the next most probable token, which is often confident-sounding text from its training data. This is why its confidence is fragile and easily swayed.

Top LLMs like Claude 3 and DeepSeek score 0% on complex Sudoku puzzles, a task humans can solve. This isn't a minor flaw but a categorical failure, exposing the transformer architecture's inability to handle constraint satisfaction problems that require backtracking and parallel reasoning, unlike its sequential, token-by-token processing.

Current LLMs abstract language into discrete tokens, losing rich information like font, layout, and spatial arrangement. A "pixel maximalist" view argues that processing visual representations of text (as humans do) is a more lossless, general approach that captures the physical manifestation of language in the world.

Large Language Models learn the structure and language of mathematical solutions from vast text data. This allows them to generate convincing explanations and steps, but they don't perform actual calculations. Their "fluency" in math-like text is different from a calculator's logical execution, leading to confident but incorrect answers.

Beyond the obvious lack of non-English training data, Large Language Models are architecturally biased. Their tokenization process, designed for English, inefficiently breaks down other languages into more fragments. This increases operational costs and reduces comprehension, creating a structural disadvantage.

When asked to analyze 100 papers, LLMs often admit they didn't complete the task. This failure stems from outcome-based training, which prioritizes a plausible-looking final output over correctly following the required process, revealing a fundamental flaw in current training paradigms.

The binary distinction between "reasoning" and "non-reasoning" models is becoming obsolete. The more critical metric is now "token efficiency"—a model's ability to use more tokens only when a task's difficulty requires it. This dynamic token usage is a key differentiator for cost and performance.