26 typical A/B testing mistakes that can lead to up to 42% annual revenue loss

Booking.com’s loss from unsuccessful experiments is 2% of annual revenue. Let’s consider this as a benchmark. What revenue loss might a less experienced team have?

When you think about very costly experimentation mistakes, the first thing that comes to mind are things like “Buy” button bugs. But it’s not the most dangerous thing though, as it’s very evident, easily recognizable, and short-term.

The worst story I heard was of a 42% of annual revenue drop as a result of deploying a feature based on false positive experiment data.

We at Conversionrate.store have developed ~7200 A/B tests for 231 clients including Microsoft …. and 72% of our first 100 experiments had mistakes we realized only 8 months after starting A/B testing.

Here are 4 common problems related to experimentation that can dramatically decrease revenue or slow down the growth:

  1. Implementation of false-positive results
  2. No A/B testing at all for critical changes
  3. Direct revenue loss from underperforming variations
  4. Not maximizing the volume and velocity of experiments

All those issues are interconnected, so let’s go through 26 typical A/B testing mistakes that we see time and time again:

  1. Hypothesis is not focused on the main bottleneck
  2. Guessing reasons behind the main bottleneck
  3. Guessing how to fix the cause of the drop-off
  4. Holding the wrong metric like conversion-to-purchase as a goal
  5. Data tracking not at least 90-97% accurate
  6. No event mapping for all elements on A and B
  7. Testing more than one hypothesis per experiment
  8. Stopping the experiment only based on statistical significance
  9. No MDE and pre-test sample size planning
  10. No QA of alternative versions after experiment is launched and no monitoring of experiment session recordings
  11. No regression QA of the control version during an experiment
  12. No QA of experiment data tracking
  13. Not eliminating the “novelty effect”

    Increase website conversion rates

  14. Implementation of false positive results
  15. No anomaly detection
  16. Outliers not cleaned up
  17. No preliminary A/A or A/A/B tests
  18. No analytics or tracking of long-term impact of implemented winning versions
  19. No in-depth post-test research and documentation of results
  20. Targeting irrelevant traffic segments together in one experiment
  21. Not checking for sample-ratio mismatch (SRM) for 100% of experiment traffic or all meaningful segments you want to compare
  22. Experiment data set not visualized
  23. Deploying winning versions to a different audience than in the experiment.
  24. Low experimentation velocity due to lack of in-house resources or absence of 100% dedicated experimentation teams
  25. Not leveraging parallel experiments when there is enough traffic
  26. Not speeding up experiment time with CUPED or similar techniques that leverage historical data on sensitivity of metrics.

Any alarm bells ringing? Even one mistake from the list can spoil experimentation results or slow down your growth rate.

Schedule a free A/B testing consultation where we can go through your experimentation process, discover its bottlenecks and consult on ways to maximize it’s volume, velocity and uplift.

Glib Hodorovskiy, co-founder -- Conversionrate.store --

Glib Hodorovskiy, co-founder Conversionrate.store

Conversionrate.store is a performance-based funnel conversion rate optimization agency that worked with 3 NASDAQ-listed clients (Microsoft, GAIA, CarID).

Schedule a consultation.