To evaluate OpenAI's GDPVal benchmark, Artificial Analysis uses Gemini 3 Pro as a judge. For complex, criteria-driven agentic tasks, this LLM-as-judge approach works well and does not exhibit the typical bias of preferring its own outputs, because the judging task is sufficiently different from the execution task.
Related Insights
Standard benchmarks fall short for multi-turn AI agents. A new approach is the 'job interview eval,' where an agent is given an underspecified problem. It is then graded not just on the solution, but on its ability to ask clarifying questions and handle changing requirements, mimicking how a human developer is evaluated.
For complex, multi-turn agentic workflows, Tasklet prioritizes a model's iterative performance over standard benchmarks. Anthropic's models are chosen based on a qualitative "vibe" of being superior over long sequences of tool use, a nuance that quantitative evaluations often miss.
The key to enabling an AI agent like Ralph to work autonomously isn't just a clever prompt, but a self-contained feedback loop. By providing clear, machine-verifiable "acceptance criteria" for each task, the agent can test its own work and confirm completion without requiring human intervention or subjective feedback.
Do not blindly trust an LLM's evaluation scores. The biggest mistake is showing stakeholders metrics that don't match their perception of product quality. To build trust, first hand-label a sample of data with binary outcomes (good/bad), then compare the LLM judge's scores against these human labels to ensure agreement before deploying the eval.
When creating an "LLM as a judge" to automate evaluations, resist the urge to use a 1-5 rating scale. This creates ambiguity (what does a 3.2 vs 3.7 mean?). Instead, force the judge to make a binary "pass" or "fail" decision. It's a more painful but ultimately more tractable and actionable way to measure quality.
The prompts for your "LLM as a judge" evals function as a new form of PRD. They explicitly define the desired behavior, edge cases, and quality standards for your AI agent. Unlike static PRDs, these are living documents, derived from real user data and are constantly, automatically testing if the product meets its requirements.
OpenAI identifies agent evaluation as a key challenge. While they can currently grade an entire task's trace, the real difficulty lies in evaluating and optimizing the individual steps within a long, complex agentic workflow. This is a work-in-progress area critical for building reliable, production-grade agents.
When testing models on the GDPVal benchmark, Artificial Analysis's simple agent harness allowed models like Claude to outperform their official web chatbot counterparts. This implies that bespoke chatbot environments are often constrained for cost or safety, limiting a model's full agentic capabilities which developers can unlock with custom tooling.
OpenAI's new GDP-val benchmark evaluates models on complex, real-world knowledge work tasks, not abstract IQ tests. This pivot signifies that the true measure of AI progress is now its ability to perform economically valuable human jobs, making performance metrics directly comparable to professional output.
Using an LLM to grade another's output is more reliable when the evaluation process is fundamentally different from the task itself. For agentic tasks, the performer uses tools like code interpreters, while the grader analyzes static outputs against criteria, reducing self-preference bias.