Klaviyo's internal "Dark Factory" system uses a team of agents to take a high-level prompt, decompose it into specs and engineering subsystems, write code, and establish API contracts. This automates the software toolchain, from prototyping to load testing, with minimal human intervention.

Boulton & Watt built an internal AI agent that processes customer interview transcripts. It maps findings to core hypotheses, highlighting supporting and contradicting evidence. This keeps the team rigorous and fact-based, counteracting natural founder bias during the discovery process.

Instead of asking customers to evaluate 50 options, use AI as a "BS layer detector" to identify the top three contenders. This saves time and budget by focusing human-led research on a pre-vetted, smaller set of choices for final validation.

Treat internal AI tools like a PR review bot with the same rigor as external products by implementing an evaluation loop. Have engineers regularly review the AI's decisions and risk scores to provide feedback, ensuring the bot's accuracy, reliability, and continuous improvement over time.

Since AI makes coding cheap, the real advantage lies in 'product taste.' Develop this by building an agent that consumes and synthesizes feedback from all sources—GitHub, Slack, Gong transcripts, and Twitter—to identify key user pains and roadmap priorities.

The temptation to use AI to rapidly generate, prioritize, and document features without deep customer validation poses a significant risk. This can scale the "feature factory" problem, allowing teams to build the wrong things faster than ever, making human judgment and product thinking paramount.

To manage non-deterministic AI products, Shopify created an internal tool where PMs grade AI-generated outputs. This creates a "ground truth" dataset of what "good" looks like, which is then used to fine-tune a separate LLM that acts as an automated quality judge for new features and updates.

Instead of relying on their lead designer for manual "vibe checks," the Braintrust team translates his qualitative feedback into quantifiable evaluation criteria. This "captures" the expert in the system, allowing his high quality bar to be applied systematically and at scale across the entire product.

Prioritize qualitative 'vibe testing' over quantitative evals in early agent development. The most crucial first step is getting the agent in front of users to see if it 'feels' right and is useful before investing in formal, scalable quality checks.