Traditional vehicle safety (e.g., Euro NCAP) used a checklist of specific test cases with binary pass/fail answers. For AI systems, this is insufficient. The new paradigm is statistical validation, where the goal is to prove reliability to a certain number of "nines" across a vast range of scenarios.

The primary barrier to adopting formal verification has been the immense cost (often 10x development time) of maintaining proofs as software changes. AI excels at this tedious and difficult task, rewriting and adapting proofs automatically, which is the key change making the practice scalable and mainstream.

By monitoring a model's internal activations during inference, safety checks can be performed with minimal overhead. Rinks claims to have reduced the compute for protecting an 8B parameter model from a 160B parameter guard model operation down to just 20M parameters—a "rounding error" that makes robust safety on edge devices finally feasible.

The act of creating a formal proof for a piece of software forces a level of rigor that surpasses even implementing it from scratch. This newfound confidence and clarity allows engineers to pursue aggressive optimizations without the fear of introducing subtle bugs, which they would otherwise avoid due to uncertainty.

While AI can generate code, the stakes on blockchain are too high for bugs, as they lead to direct financial loss. The solution is formal verification, using mathematical proofs to guarantee smart contract correctness. This provides a safety net, enabling users and AI to confidently build and interact with financial applications.

A major hurdle for formal methods is the effort required to write proofs. Generative AI is becoming capable of producing proofs in formal languages like Lean, which can then be automatically verified by a machine. This could make verified software development scalable for the first time.

Formal verification, the process of mathematically proving software correctness, has been too complex for widespread use. New AI models can now automate this, allowing developers to build systems with mathematical guarantees against certain bugs—a huge step for creating trust in high-stakes financial software.

Instead of supervising an AI's hidden thought process, we can demand it produces a 'certificate of reasoning'—a checkable proof—along with its output. This could include citations or sensitivity analyses, shifting verification from observing the process to checking the provided proof.

For critical processes in regulated industries, standard AI model evaluations ("evals") are insufficient. Enterprises like UBS are pushing for research into mathematical proofs to formally verify that AI agents behave correctly across multiple tasks, establishing a much higher standard of trust and safety.