Languages like Lean allow mathematical proofs to be automatically verified. This provides a perfect, binary reward signal (correct/incorrect) for a reinforcement learning agent. It transforms the abstract art of mathematics into a well-defined environment, much like a game of Go, that an AI can be trained to master.
Related Insights
Generative AI can produce the "miraculous" insights needed for formal proofs, like finding an inductive invariant, which traditionally required a PhD. It achieves this by training on vast libraries of existing mathematical proofs and generalizing their underlying patterns, effectively automating the creative leap needed for verification.
A Rice PhD showed that training a vision model on a game like Snake, while prompting it to see the game as a math problem (a Cartesian grid), improved its math abilities more than training on math data directly. This highlights how abstract, game-based training can foster more generalizable reasoning.
To reliably translate a natural language policy into formal logic, Amazon's system generates multiple translations using an LLM. It then employs a theorem prover to verify these translations are logically equivalent. Mismatches trigger a clarification loop with the user, ensuring the final specification is correct before checking an agent's work.
Andrej Karpathy's 'Software 2.0' framework posits that AI automates tasks that are easily *verifiable*. This explains the 'jagged frontier' of AI progress: fields like math and code, where correctness is verifiable, advance rapidly. In contrast, creative and strategic tasks, where success is subjective and hard to verify, lag significantly behind.
While AI can generate code, the stakes on blockchain are too high for bugs, as they lead to direct financial loss. The solution is formal verification, using mathematical proofs to guarantee smart contract correctness. This provides a safety net, enabling users and AI to confidently build and interact with financial applications.
In domains like coding and math where correctness is automatically verifiable, AI can move beyond imitating humans (RLHF). Using pure reinforcement learning, or "experiential learning," models learn via self-play and can discover novel, superhuman strategies similar to AlphaGo's Move 37.
Reinforcement Learning with Human Feedback (RLHF) is a popular term, but it's just one method. The core concept is reinforcing desired model behavior using various signals. These can include AI feedback (RLAIF), where another AI judges the output, or verifiable rewards, like checking if a model's answer to a math problem is correct.
AI and formal methods have been separate fields with opposing traits: AI is flexible but untrustworthy, while formal methods offer guarantees but are rigid. The next frontier is combining them into neurosymbolic systems, creating a "peanut butter and chocolate" moment that captures the best of both worlds.
The transition from supervised learning (copying internet text) to reinforcement learning (rewarding a model for achieving a goal) marks a fundamental breakthrough. This method, used in Anthropic's Opus 3 model, allows AI to develop novel problem-solving capabilities beyond simple data emulation.
We perceive complex math as a pinnacle of intelligence, but for AI, it may be an easier problem than tasks we find trivial. Like chess, which computers mastered decades ago, solving major math problems might not signify human-level reasoning but rather that the domain is surprisingly susceptible to computational approaches.
