Before building an AI agent, product managers must first create an evaluation set and scorecard. This 'eval-driven development' approach is critical for measuring whether training is improving the model and aligning its progress with the product vision. Without it, you cannot objectively demonstrate progress.

Building for current AI models is shortsighted, while building for a year ahead is too speculative. OpenAI's product teams target the anticipated model capabilities of the next quarter to stay relevant without being premature.

Product management at OpenAI is defined by ambiguity because the full capabilities and emergent behaviors of the next model are unknown even to the team building it. This requires PMs to maintain extremely flexible roadmaps that can adapt quickly as research breakthroughs occur.

Anthropic's product managers on the research team spec out requirements for each new AI model, defining what it should be good at (e.g., coding, knowledge work). This product development discipline is applied to the inherently unpredictable process of "growing" a model, bridging the gap between research and user needs.

Instead of designing for current AI limitations, the team projects model improvements six months forward and builds interfaces for that future state. This framework guided the creation of their agent-first products, assuming capabilities would eventually catch up to the design.

Contrary to assumption, the design process at OpenAI isn't about planning for a distant future. It's a fast-paced environment where designers work in close concert with the latest research advancements, adapting to new capabilities as they emerge.

By embedding product teams directly within the research organization, Google creates a tight feedback loop. Instead of receiving models "over the wall," product and research teams co-develop them, aligning technical capabilities with customer needs from the start.

The PM role will expand beyond leveraging off-the-shelf AI. They will be responsible for creating and training specialized AI agents. This involves instilling agents with deep, company-specific knowledge of business models, customers, and strategy, just as they would onboard a new human team member.

Traditional product management separates customer problem discovery from technical implementation. In AI, this model fails. Winning teams must deeply understand the nuanced, 'jagged edge' of what models can and can't do, building products that bridge that specific, shifting capability frontier with customer problems.