In large companies, a culture of A/B testing every decision can become a crutch that stifles innovation and speed. It leads to risk aversion and organizational lethargy, as teams lose the muscle for making convicted, gut-based decisions informed by qualitative customer feedback.

The traditional "test and learn" mantra is flawed because teams often start with a weak set of creative variants. By using predictive AI to generate a diverse but pre-vetted, high-performance set of options, marketers can ensure their tests are more meaningful and aren't just optimizing a bad strategy.

When testing copy like titles or subject lines, change only a single modifier word (e.g., add "Quick Fix" to "HR Guide"). This isolates the variable, providing clear learnings about what resonates with your audience, unlike testing two completely different sentences where the "why" is unclear.

Personal biases and preferences should not dictate marketing strategy. A marketer might dislike pop-ups or emojis, but that doesn't mean their target audience feels the same. The most valuable tests often involve tactics that challenge a marketer's own assumptions about what works.

Don't attempt traditional A/B testing on a low-traffic website; the results will be statistically invalid. Instead, use qualitative user testing methods like preference tests. This approach provides directional data to guide decisions, which is far more reliable than guesswork or a flawed A/B test.

For statistically significant A/B test results on major changes like text vs. design, don't rely on a single send. Test within an automated series (e.g., a welcome flow) and collect data for an extended period, like a full quarter, to remove seasonality and ensure a healthy sample size.

For a rapidly scaling brand, optimizing for small, single-digit percentage gains is a waste of time. Prioritize tests that have the potential for massive, double-digit improvements. If a test doesn't show a clear, significant winner quickly, abandon it and move to the next big idea.

Despite mature backtesting frameworks, Intercom repeatedly sees promising offline results fail in production. The "messiness of real human interaction" is unpredictable, making at-scale A/B tests essential for validating AI performance improvements, even for changes as small as a tenth of a percentage point.

The potential upside of a successful marketing test is limitless, while the downside of a failure is capped and brief. If all your tests are winning, you are likely only testing obvious optimizations and missing out on bigger, game-changing breakthroughs that come from more ambitious experiments.