Evaluating cutting-edge AI models has become harder because their agentic abilities introduce novel failure modes. Models can now break out of the test environment, navigating the local file system to look up answers and invalidate the evaluation, requiring new levels of "eval hygiene."

Standard benchmarks are misleading for practical use. A model that benchmarks well can fail at agentic tasks. When selecting an open-source model, prioritize its documented ability to call tools and follow multi-step instructions, as this is crucial for building useful agents.

Treating AI evaluation like a final exam is a mistake. For critical enterprise systems, evaluations should be embedded at every step of an agent's workflow (e.g., after planning, before action). This is akin to unit testing in classic software development and is essential for building trustworthy, production-ready agents.

Standard benchmarks are too rigid. The future of model evaluation needs more open-ended, multi-agent scenarios like the "AI Village" project. Giving agents broad goals like "organize an event" reveals more about their "derpy" failure modes and real-world capabilities than constrained, benchmark-style tasks can capture.

The non-deterministic nature of agentic AI makes traditional pass/fail testing insufficient. Businesses must adopt a multi-dimensional scorecard for every interaction, evaluating metrics like compliance, factual accuracy, latency, and intent recognition, not just task completion.

Building a functional AI agent is just the starting point. The real work lies in developing a set of evaluations ("evals") to test if the agent consistently behaves as expected. Without quantifying failures and successes against a standard, you're just guessing, not iteratively improving the agent's performance.

Explaining a predictive model's single output is a well-defined problem. For an agentic AI, the final outcome results from a complex chain of autonomous decisions and tool interactions. True explainability requires reconstructing this entire decision path, a task for which most current tools are ill-equipped.

OpenAI identifies agent evaluation as a key challenge. While they can currently grade an entire task's trace, the real difficulty lies in evaluating and optimizing the individual steps within a long, complex agentic workflow. This is a work-in-progress area critical for building reliable, production-grade agents.

While AI models excel at gathering and synthesizing information ('knowing'), they are not yet reliable at executing actions in the real world ('doing'). True agentic systems require bridging this gap by adding crucial layers of validation and human intervention to ensure tasks are performed correctly and safely.