26 typical A/B testing mistakes that can lead to up to 42% annual revenue loss

Booking.com’s loss from unsuccessful experiments is 2% of annual revenue. Let’s consider this as a benchmark. What revenue loss might a less experienced team have?

When you think about very costly experimentation mistakes, the first thing that comes to mind are things like “Buy” button bugs. But it’s not the most dangerous thing though, as it’s very evident, easily recognizable, and short-term.

The worst story I heard was of a 42% of annual revenue drop as a result of deploying a feature based on false positive experiment data.

We at Conversionrate.store have developed ~7200 A/B tests for 231 clients including Microsoft …. and 72% of our first 100 experiments had mistakes we realized only 8 months after starting A/B testing.

Here are 4 common problems related to experimentation that can dramatically decrease revenue or slow down the growth:

1. Implementation of false-positive results
2. No A/B testing at all for critical changes
3. Direct revenue loss from underperforming variations
4. Not maximizing the volume and velocity of experiments

All those issues are interconnected, so let’s go through 26 typical A/B testing mistakes that we see time and time again:

1. Hypothesis is not focused on the main bottleneck
2. Guessing reasons behind the main bottleneck
3. Guessing how to fix the cause of the drop-off
4. Holding the wrong metric like conversion-to-purchase as a goal
5. Data tracking not at least 90-97% accurate
6. No event mapping for all elements on A and B
7. Testing more than one hypothesis per experiment
8. Stopping the experiment only based on statistical significance
9. No MDE and pre-test sample size planning
10. No QA of alternative versions after experiment is launched and no monitoring of experiment session recordings
11. No regression QA of the control version during an experiment
12. No QA of experiment data tracking
13. Not eliminating the “novelty effect”

14. Implementation of false positive results
15. No anomaly detection
16. Outliers not cleaned up
17. No preliminary A/A or A/A/B tests
18. No analytics or tracking of long-term impact of implemented winning versions
19. No in-depth post-test research and documentation of results
20. Targeting irrelevant traffic segments together in one experiment
21. Not checking for sample-ratio mismatch (SRM) for 100% of experiment traffic or all meaningful segments you want to compare
22. Experiment data set not visualized
23. Deploying winning versions to a different audience than in the experiment.
24. Low experimentation velocity due to lack of in-house resources or absence of 100% dedicated experimentation teams
25. Not leveraging parallel experiments when there is enough traffic
26. Not speeding up experiment time with CUPED or similar techniques that leverage historical data on sensitivity of metrics.

Any alarm bells ringing? Even one mistake from the list can spoil experimentation results or slow down your growth rate.

Schedule a free A/B testing consultation where we can go through your experimentation process, discover its bottlenecks and consult on ways to maximize it’s volume, velocity and uplift.