While evals involve testing, their purpose isn't just to report bugs (information), like traditional QA. For an AI PM, evals are a core tool to actively shape and improve the product's behavior and performance (transformation) by iteratively refining prompts, models, and orchestration layers.

With AI agents automating raw code generation, an engineer's role is evolving beyond pure implementation. To stay valuable, engineers must now cultivate a deep understanding of business context and product taste to know *what* to build and *why*, not just *how*.

The most impactful quality metrics are not internal measures like bug counts but those directly linked to customer and business outcomes. QA professionals increase their influence by framing their findings in terms of business impact, financial exposure, and customer risk.

To get product management buy-in for technical initiatives like refactoring or scaling, engineering leadership is responsible for translating the work into clear business or customer value. Instead of just stating the technical need, explain how it enables faster feature development or access to a larger customer base.

With AI generating code, a developer's value shifts from writing perfect syntax to validating that the system works as intended. Success is measured by outcomes—passing tests and meeting requirements—not by reading or understanding every line of the generated code.

Shift the definition of "done" from "code checked in" to "logged in as the user and verified the feature works as intended." This simple directive forces engineers to engage with the product from a user's perspective, fostering ownership and higher quality work.

When handed a specific solution to build, don't just execute. Reverse-engineer the intended customer behavior and outcome. This creates an opportunity to define better success metrics, pressure-test the underlying problem, and potentially propose more effective solutions in the future.

AI excels at generating code, making that task a commodity. The new high-value work for engineers is "verification”—ensuring the AI's output is not just bug-free, but also valuable to customers, aligned with business goals, and strategically sound.

AI tools can dramatically accelerate test execution but lack the contextual understanding to interpret results or assess business risk. An effective hybrid model has humans own the 'what' and 'why' (sense-making) while AI handles the 'how fast' (execution).