The conventional, sequential stages of software development (design, code, test, review) are becoming obsolete. AI agents merge these steps into a single, iterative loop driven by user intent. This isn't a 10x improvement on the existing workflow; it's a fundamental paradigm shift that makes the entire traditional process a relic.

Before building an AI agent, product managers must first create an evaluation set and scorecard. This 'eval-driven development' approach is critical for measuring whether training is improving the model and aligning its progress with the product vision. Without it, you cannot objectively demonstrate progress.

Traditional software engineering valued meticulous upfront planning to avoid costly coding and debugging cycles. Newman argues that with AI agents, the cost of building and iterating is so low that the old "measure twice, cut once" philosophy is obsolete. The superior modern approach is to build quickly, even incorrectly, and rapidly iterate.

Treating AI evaluation like a final exam is a mistake. For critical enterprise systems, evaluations should be embedded at every step of an agent's workflow (e.g., after planning, before action). This is akin to unit testing in classic software development and is essential for building trustworthy, production-ready agents.

Building reliable AI agents requires a developer mindset shift. The most critical task is not writing the agent's code but creating robust evaluations ('evals') that define and verify the desired business outcome. This makes a test-driven development approach non-negotiable for enterprise AI.

Move beyond manual agent improvement by creating an automated loop. In this process, an agent runs, its performance is evaluated, failures are identified, and another process suggests and implements code fixes. This creates a foundation for self-improving systems.

Don't start building evaluations from a blank slate. Use an AI agent to analyze your production traces and automatically generate a baseline 'vibe eval.' This initial evaluation won't be perfect, but it provides a starting point for refinement and accelerates the improvement loop.

Notion treats its entire evaluation process as a coding agent problem. The system is designed for an agent to download a dataset, run an eval, identify a failure, debug the issue, and implement a fix, all within an automated loop. This turns quality assurance into a meta-problem for agents to solve.

Building a functional AI agent is just the starting point. The real work lies in developing a set of evaluations ("evals") to test if the agent consistently behaves as expected. Without quantifying failures and successes against a standard, you're just guessing, not iteratively improving the agent's performance.