Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

James McAulay built rules and loops so Claude asks whether a running growth experiment can be concluded, what was learned, and records the answer. He links this to his first company, where undocumented experiments meant new hires kept rediscovering old failures. The system also flags experiments that are stale or overdue.

Related Insights

Encourage experimentation with AI, but mitigate risk by giving every pilot a clear budget limit ('a leash') and a deadline ('a clock'). This framework allows for innovation while ensuring underperforming projects are killed quickly, successful ones are funded, and great ones are scaled.

To accelerate organizational learning in AI, incentivize the sharing of failures. A Fortune 500 company gives employees redeemable points for sharing use cases, but offers *extra points* for detailing a failed experiment and the resulting lesson. This normalizes failure and prevents others from repeating the same mistakes.

Anthropic is developing a system called "CASH" to automate growth work. It uses Claude to identify opportunities, build features (like copy changes), test them, and analyze results. The system is already delivering results comparable to a junior PM and is expected to handle increasingly complex experiments.

Kieran Klaassen's "Compound Engineering" philosophy involves planning, working, assessing, and then codifying learnings. This feedback loop teaches the AI what it did wrong, ensuring it won't repeat the same mistakes and making it progressively better with each use.

To improve AI results over time, create a feedback loop. After running marketing experiments, use an MCP to save the results and analysis as a file within your Idea Browser project. This creates a compounding knowledge base, giving the AI richer context for making more informed strategic decisions in the future.

Instead of codebases becoming harder to manage over time, use an AI agent to create a "compounding engineering" system. Codify learnings from each feature build—successful plans, bug fixes, tests—back into the agent's prompts and tools, making future development faster and easier.

When an AI tool makes a mistake, treat it as a learning opportunity for the system. Ask the AI to reflect on why it failed, such as a flaw in its system prompt or tooling. Then, update the underlying documentation and prompts to prevent that specific class of error from happening again in the future.

Standard LLMs often validate ideas to be helpful. Implement a structured "viability gate" skill with clear evaluation criteria (e.g., problem clarity, competition) designed to explicitly recommend abandoning unpromising projects, saving valuable time and resources.

Don't just assume a new AI workflow is better. Treat internal process changes with the same rigor as product features. Apply a hypothesis-driven framework to how your team operates, experimenting with new AI tools and methods, and validating whether they actually improve outcomes before committing to them.

Instead of individual use, engineers on OpenAI's growth team created a shared, reusable Codex "skill" for the entire experiment review process. By pointing it to a Statsig experiment, the skill writes hypotheses, monitors progress, and generates a post-mortem with recommendations.

AI experiment system forces closure and documentation | RiffOn