Companies with valuable proprietary data should not license it away. A better strategy to guide foundation model development is to keep the data private but release public benchmarks and evaluations based on it. This incentivizes LLM providers to train their models on the specific tasks you care about, improving their performance for your product.

The company provides public benchmarks for free to build trust. It monetizes by selling private benchmarking services and subscription-based enterprise reports, ensuring AI labs cannot pay for better public scores and thus maintaining objectivity.

Public leaderboards like LM Arena are becoming unreliable proxies for model performance. Teams implicitly or explicitly "benchmark" by optimizing for specific test sets. The superior strategy is to focus on internal, proprietary evaluation metrics and use public benchmarks only as a final, confirmatory check, not as a primary development target.

To ensure they are testing the same models available to the public, they register anonymous accounts to run evals. This prevents labs from providing specially tuned private endpoints that perform better than their publicly available APIs, thereby maintaining the integrity of their independent analysis.

LM Arena, known for its public AI model rankings, generates revenue by selling custom, private evaluation services to the same AI companies it ranks. This data helps labs improve their models before public release, but raises concerns about a "pay-to-play" dynamic that could influence public leaderboard performance.

Arena differentiates from competitors like Artificial Analysis by evaluating models on organic, user-generated prompts. This provides a level of real-world relevance and data diversity that platforms using pre-generated test cases or rerunning public benchmarks cannot replicate.

To maintain independence and trust, their public benchmarks are free and cannot be influenced by payments. The company generates revenue by selling detailed reports and insight subscriptions to enterprises, and by conducting private, custom benchmarking for AI companies, separating their public good from their commercial offerings.

LM Arena's $1.7B valuation stems from its innovative flywheel: it attracts millions of users to a simple "pick your favorite AI" game, generating data that becomes the industry's most trusted leaderboard. This forces major AI labs to pay for evaluations, turning a user engagement loop into a powerful marketing and revenue engine.

They provide extensive free benchmarks to build credibility and community trust. Monetization comes from enterprise subscriptions for deeper insights and private, custom benchmarking for AI companies, ensuring the public data remains independent.