The proliferation of AI leaderboards incentivizes companies to optimize models for specific benchmarks. This creates a risk of "acing the SATs" where models excel on tests but don't necessarily make progress on solving real-world problems. This focus on gaming metrics could diverge from creating genuine user value.

Static benchmarks are easily gamed. Dynamic environments like the game Diplomacy force models to negotiate, strategize, and even lie, offering a richer, more realistic evaluation of their capabilities beyond pure performance metrics like reasoning or coding.

General Intuition's first commercial use case for its human-like AI agents isn't a consumer product, but a B2B tool for game developers. High-quality bots are crucial for retaining players by ensuring full lobbies during off-peak hours when human player numbers are low, providing a clear, revenue-generating entry point for their sophisticated AI.

Startups like Cognition Labs find their edge not by competing on pre-training large models, but by mastering post-training. They build specialized reinforcement learning environments that teach models specific, real-world workflows (e.g., using Datadog for debugging), creating a defensible niche that larger players overlook.

Instead of building AI models, a company can create immense value by being 'AI adjacent'. The strategy is to focus on enabling good AI by solving the foundational 'garbage in, garbage out' problem. Providing high-quality, complete, and well-understood data is a critical and defensible niche in the AI value chain.

When Good Star Labs streamed their AI Diplomacy game on Twitch, it attracted 50,000 viewers from the gaming community. Watching AIs make mistakes, betray allies, and strategize made the technology more relatable and less intimidating, helping to bridge the gap between AI experts and the general public.

AI evaluation shouldn't be confined to engineering silos. Subject matter experts (SMEs) and business users hold the critical domain knowledge to assess what's "good." Providing them with GUI-based tools, like an "eval studio," is crucial for continuous improvement and building trustworthy enterprise AI.

Companies like OpenAI and Anthropic are spending billions creating simulated enterprise apps (RL gyms) where human experts train AI models on complex tasks. This has created a new, rapidly growing "AI trainer" job category, but its ultimate purpose is to automate those same expert roles.

A niche, services-heavy market has emerged where startups build bespoke, high-fidelity simulation environments for large AI labs. These deals command at least seven-figure price tags and are critical for training next-generation agentic models, despite the customer base being only a few major labs.