LLMs shine when acting as a 'knowledge extruder'—shaping well-documented, 'in-distribution' concepts into specific code. They fail when the core task is novel problem-solving where deep thinking, not code generation, is the bottleneck. In these cases, the code is the easy part.

Formal proof systems like Lean provide a unique training ground for LLMs. Unlike natural language reasoning, a proof's correctness can be programmatically verified. This creates a strong reward signal for training long-horizon planning and coherence, skills that can generalize to other tasks.

The argument that LLMs are just "stochastic parrots" is outdated. Current frontier models are trained via Reinforcement Learning, where the signal is not "did you predict the right token?" but "did you get the right answer?" This is based on complex, often qualitative criteria, pushing models beyond simple statistical correlation.

LLMs excel at coding because internet data (e.g., GitHub) provides complete source code, dependencies, and reasoning. In contrast, mathematical texts online are often just condensed summaries or final proofs, lacking the step-by-step process. This makes it harder for models to learn mathematical reasoning from pre-training alone.

Probabilistic models excel at text because a sentence near the 'perfect' answer is usually still valid. In contrast, math is a 'hostile space' where the right answer is surrounded by wrong ones. A small deviation (e.g., 4.1 instead of 4) results in a completely incorrect output, explaining the challenge for LLMs.

Large Language Models learn the structure and language of mathematical solutions from vast text data. This allows them to generate convincing explanations and steps, but they don't perform actual calculations. Their "fluency" in math-like text is different from a calculator's logical execution, leading to confident but incorrect answers.

When asked to analyze 100 papers, LLMs often admit they didn't complete the task. This failure stems from outcome-based training, which prioritizes a plausible-looking final output over correctly following the required process, revealing a fundamental flaw in current training paradigms.

OpenPipe's 'Ruler' library leverages a key insight: GRPO only needs relative rankings, not absolute scores. By having an LLM judge stack-rank a group of agent runs, one can generate effective rewards. This approach works phenomenally well, even with weaker judge models, effectively solving the reward assignment problem.

Instead of pinpointing which specific action led to a good outcome, the GRPO algorithm solves the credit assignment problem with a simple heuristic: it assumes rare tokens in a high-scoring output were responsible and upweights all of them. This "unsatisfying" but practical approach works surprisingly well.