The military's primary incentive is to use weapons that are effective and reliable, as soldiers' lives depend on it. This inherent conservatism acts as a strong filter against deploying unproven or unpredictable AI systems, making them slower, not faster, to adopt bleeding-edge technology in life-or-death situations.

The strategy's focus on AI simulation acknowledges a key risk: AI systems can develop winning tactics by exploiting unrealistic aspects of a simulation. If simulation physics or capabilities don't perfectly match reality, these AI-derived strategies could fail catastrophically when deployed.

There's a significant gap between AI performance in simulated benchmarks and in the real world. Despite scoring highly on evaluations, AIs in real deployments make "silly mistakes that no human would ever dream of doing," suggesting that current benchmarks don't capture the messiness and unpredictability of reality.

Standardized benchmarks for AI models are largely irrelevant for business applications. Companies need to create their own evaluation systems tailored to their specific industry, workflows, and use cases to accurately assess which new model provides a tangible benefit and ROI.

Standard benchmarks are too rigid. The future of model evaluation needs more open-ended, multi-agent scenarios like the "AI Village" project. Giving agents broad goals like "organize an event" reveals more about their "derpy" failure modes and real-world capabilities than constrained, benchmark-style tasks can capture.

Contrary to the 'killer robots' narrative, the military is cautious when integrating new AI. Because system failures can be lethal, testing and evaluation standards are far stricter than in the commercial sector. This conservatism is driven by warfighters who need tools to work flawlessly.

Smack Technologies argues that general-purpose LLMs fail in military strategy because they rely on historical labeled data. For novel, high-stakes conflicts, a different approach like deep reinforcement learning is required, training models within physics-grounded simulations of potential future battlefields.

Shield AI identifies the key problem in defense tech as simultaneously achieving high performance, ensuring high levels of safety and assurance, and maintaining rapid development cycles. Historically, systems had to trade these off, but modern defense requires solving for all three concurrently.

Contrary to popular belief, military procurement involves some of the most rigorous safety and reliability testing. Current generative AI models, with their inherent high error rates, fall far short of these established thresholds that have long been required for defense systems.