Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

The new frontier for product management is diving deep into AI systems' operational data. According to Arize's CPO, PMs who regularly analyze traces and evaluations to understand agent behavior are far ahead of their peers and represent the top 1% of the field.

Related Insights

As AI automates generalist PM tasks like documentation and context sharing, the role is evolving. The new path to value is specialization. PMs should identify their passion—be it data, design, or prototyping—and master the corresponding AI tools to develop deep, defensible expertise.

Unlike traditional software where UX can be pre-assessed, AI products are inherently unpredictable. The CEO of Braintrust argues that this makes observability critical. Companies must monitor real-world user interactions to capture failures and successes, creating a data flywheel for rapid improvement.

AI's rapid capability growth makes top-down product specs obsolete. Product Managers now work bottoms-up with engineers, prototyping and even checking in code using AI tools. This blurs traditional roles, shifting the PM's focus to defining high-level customer needs and evaluating outcomes rather than prescribing features.

AI won't replace product managers but will elevate their role. PMs will shift from executing tasks like financial forecasting to managing a team of specialized AI agents, forcing them to focus on high-level strategy and assumption-checking.

The PM role is shifting to that of a 'product builder.' Instead of manually sifting through data, they can use AI agents to scrape sources like Gong, Slack, and Intercom. This provides an aggregated 'voice of the customer' and a data-backed strategy in minutes, not weeks.

In AI development, trace analysis is a point of tension. Product Managers should become fluent enough to ask intelligent questions and participate in debugging. However, they should avoid owning the process or tooling, respecting it as engineering's domain to maintain a healthy division of labor.

The rise of AI tools isn't replacing the PM role, but transforming it. PMs who embrace an "AI-enhanced" workflow for research, docs, and prototyping will gain a massive productivity advantage, ultimately displacing those who stick to traditional methods.

In traditional product management, data was for analysis. In AI, data *is* the product. PMs must now deeply understand data pipelines, data health, and the critical feedback loop where model outputs are used to retrain and improve the product itself, a new core competency.

Counterintuitively, AI's greatest value for product managers comes from ingesting and synthesizing vast amounts of context—customer calls, data, internal documents—rather than just generating artifacts like PRDs. Superior context is the foundation for high-leverage decisions that multiply a company's output.

Reviewing user interaction data is the highest ROI activity for improving an AI product. Instead of relying solely on third-party observability tools, high-performing teams build simple, custom internal applications. These tools are tailored to their specific data and workflow, removing all friction from the process of looking at and annotating traces.