Algorithms like GRPO are powerful but require parallel rollouts in a reproducible environment. Building and maintaining these high-fidelity sandboxes, complete with realistic data and failure modes, is the hardest part of implementing RL today and a significant barrier for most companies.

Unlike language models trained on the internet, AI for materials science overcomes data scarcity and unreliability (e.g., conflicting literature) with a closed loop. The system actively directs experiments, analyzes grounded results for patterns, and uses that new data to drive the next cycle.

Moving beyond simulation, Genesis uses a cycle where their AI proposes molecules, a pharma partner synthesizes and tests them in a wet lab, and the experimental outcomes are used as feedback to retrain the generative model. This is akin to RLHF but with physical experiments.

Jerry Tworek, a self-described "RL maximalist," found that scaling RL at OpenAI improved benchmarks but failed to solve real-world problems. The training data and evals were a closed loop, disconnected from the messy distribution of real user tasks, necessitating models that can learn at test time.

Quilter avoids the intractability of training an RL agent on every minute detail of circuit board design. Instead, they structure the environment to present the agent with key, high-level decisions (e.g., "go clockwise or counter-clockwise"), drastically reducing the search space and making learning feasible.

Unlike many AI fields obsessed with compute, the primary bottleneck in materials discovery is the speed and cost of running physical experiments. Progress depends on experimental throughput, not just bigger models or more GPUs.

Mixing precursor powders to attempt new material formulations is relatively straightforward, but characterizing the resulting synthesis is the core operational bottleneck. Without automated systems to quickly interpret complex diffraction patterns and determine what was formed, high-throughput experimentation yields little value. Periodic Labs concentrates engineering on automated characterization and simulation feedback to rapidly direct subsequent experimental rounds.

Quilter's RL agent gets fast feedback using a three-tiered reward system. It starts with cheap geometric rules, moves to faster quasi-static physics approximations, and only finally uses expensive full-wave simulations. This provides rapid, conservative feedback essential for efficient training.

Mineral refineries are complex systems with up to a thousand control variables. A change in one part of the circuit may not cascade through the system for 24-48 hours. This high latency, combined with recycled material flows, makes optimization incredibly difficult for human operators.