Observed AI mathematical breakthroughs operate largely within the convex hull of existing knowledge, recombining recognized results rather than creating novel frameworks. Generating genuinely surprising hypotheses or proposing the next foundational open question remains an unresolved challenge for automated systems.

Current AI models are powerful at applying a vast library of known mathematical techniques to solve problems. However, they fall short in the more abstract, "fuzzy" areas of developing new theories, intuitions, and non-rigorous philosophies that often drive major breakthroughs in the field.

Even with vast training data, current AI models are far less sample-efficient than humans. This limits their ability to adapt and learn new skills on the fly. They resemble a perpetual new hire who can access information but lacks the deep, instinctual learning that comes from experience and weight updates.

A critical gap in AI is that systems cannot yet learn a model of their world from raw experience and then use that model for high-level planning. Successes like AlphaGo relied on a pre-defined model (the game rules), but true intelligence requires discovering these models independently.

The current focus on pre-training AI with specific tool fluencies overlooks the crucial need for on-the-job, context-specific learning. Humans excel because they don't need pre-rehearsal for every task. This gap indicates AGI is further away than some believe, as true intelligence requires self-directed, continuous learning in novel environments.

AI models are powerful "coherence machines," logically extending existing information. However, they cannot perform "abduction"—making imaginative leaps to the best explanation beyond current evidence, as innovators like Darwin did. This uniquely human skill is the engine of breakthrough innovation and where companies should focus training.

RAG systems are limited to direct retrieval and can't make spontaneous, abstract connections. This human-like ability to notice related but unasked-for concepts can only emerge from knowledge internalized within model weights, forming an associative memory.

The "memory" feature in today's LLMs is a convenience that saves users from re-pasting context. It is far from human memory, which abstracts concepts and builds pattern recognition. The true unlock will be when AI develops intuitive judgment from past "experiences" and data, a much longer-term challenge.

A fundamental misunderstanding is that AI learns from each interaction. It doesn't. Models are trained, but each new prompt is a fresh start, like 'Groundhog Day.' They operate on syntactic patterns without building semantic understanding or memory, which explains their inconsistent responses.