Companies with valuable proprietary data should not license it away. A better strategy to guide foundation model development is to keep the data private but release public benchmarks and evaluations based on it. This incentivizes LLM providers to train their models on the specific tasks you care about, improving their performance for your product.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

To ensure AI labs don't provide specially optimized private endpoints for evaluation, the firm creates anonymous accounts to test the same public models everyone else uses. This "mystery shopper" policy maintains the integrity and independence of their results.

LM Arena, known for its public AI model rankings, generates revenue by selling custom, private evaluation services to the same AI companies it ranks. This data helps labs improve their models before public release, but raises concerns about a "pay-to-play" dynamic that could influence public leaderboard performance.

Arena differentiates from competitors like Artificial Analysis by evaluating models on organic, user-generated prompts. This provides a level of real-world relevance and data diversity that platforms using pre-generated test cases or rerunning public benchmarks cannot replicate.

To maintain independence and trust, their public benchmarks are free and cannot be influenced by payments. The company generates revenue by selling detailed reports and insight subscriptions to enterprises, and by conducting private, custom benchmarking for AI companies, separating their public good from their commercial offerings.

Instead of gating its valuable review data like traditional analyst firms, G2 strategically chose to syndicate it and make it available to LLMs. This ensures G2 remains a trusted, cited source within AI-generated answers, maintaining brand influence and relevance where buyers are now making decisions.

To ensure they're testing publicly available models, Artificial Analysis creates anonymous accounts to run benchmarks without the provider's knowledge. Labs agree to this policy because it guarantees fairness and prevents their competitors from receiving special treatment or manipulating results, creating a stable, trusted equilibrium.

The founders built the tool because they needed independent, comparative data on LLM performance vs. cost for their own legal AI startup. It only became a full-time company after its utility grew with the explosion of new models, demonstrating how solving a personal niche problem can address a wider market need.