We scan new podcasts and send you the top 5 insights daily.
Jim Lanzone says Yahoo's huge audience lets teams test hunches, especially with AI. But he warns product teams can get addicted to testing and want 200 experiments at once, when they need enough passion to lead with a point of view. Testing should refine conviction, not replace it.
Launching experiments without prior customer interviews or market analysis is a waste of resources. The most effective experiments are designed to answer specific questions that arise from a solid research foundation, not to substitute for it.
In large companies, a culture of A/B testing every decision can become a crutch that stifles innovation and speed. It leads to risk aversion and organizational lethargy, as teams lose the muscle for making convicted, gut-based decisions informed by qualitative customer feedback.
While useful, A/B tests are often misused by management to avoid making a judgment call. If a test result is 51% vs. 49%, it's not a clear directive; it's a call for human leadership and conviction, which tests can never replace.
Most A/B tests are just guesswork based on best practices. A better approach is to use emotional targeting research to understand the 'why' behind user behavior. This allows you to form strategic hypotheses that lead to meaningful, not random, optimizations.
For a rapidly scaling brand, optimizing for small, single-digit percentage gains is a waste of time. Prioritize tests that have the potential for massive, double-digit improvements. If a test doesn't show a clear, significant winner quickly, abandon it and move to the next big idea.
Founders with low trial volume often mistakenly try to A/B test small changes. With insufficient data, such tests are meaningless. Instead, they should focus on making big, obvious improvements based on gut feel and qualitative feedback. At this stage, the goal isn't optimization; it's finding significant wins that don't require statistical validation.
The potential upside of a successful marketing test is limitless, while the downside of a failure is capped and brief. If all your tests are winning, you are likely only testing obvious optimizations and missing out on bigger, game-changing breakthroughs that come from more ambitious experiments.
Traditional A/B testing is fundamentally broken for modern marketing. It's a manual process that doesn't scale, its single-variable nature prevents deep, meaningful tests, and human selection of what to test introduces significant confirmation bias, limiting true discovery and innovation.
Unlike stable markets requiring deep theoretical strategy, the emergent nature of AI demands PMs shift from writing exhaustive docs to designing rapid, empirical tests focused on a single, core hypothesis.
When data from split tests is ambiguous, let your genuine enthusiasm for a particular customer segment guide your decision. This emotional investment translates into a better product and a more resilient business strategy.