Get 3 CRO hypothesis

Case study for Moonmagic.com:

a 9% uplift of ARPU as a result of introducing a cross-sell carousel in the slide-in cart

Summary

We achieved a 9% uplift of ARPU (CR uplift of 4% and AOV uplift of 9.5%) by introducing a cross-sell carousel in the slide-in cart.

 

Overview

Moonmagic.com is a fast growing online jewelry store that has 430K sessions per month (according to Similarweb). It sells rings, necklaces, earrings and other products in this category. 

Our task was to increase the revenue per user (ARPU) metric by growing the conversion rate (CR) and average order value (AOV).

 

Research

We looked at the revenue breakdown to identify key revenue drivers. We found that around 25% of buyers purchase more than one product per transaction. We concluded that users have a significant propensity to purchase more than one item per transaction.

So the idea at this point was to encourage purchases of multiple items in each transaction.

As a result of a heuristic UX analysis we found real estate in the cart to test a cross-sell carousel. 

Next, we needed a recommendations engine to use in our cross-sell carousel. A data-driven recommendations engine would work best for this purpose, so we decided to create one. We created a regression matrix to identify various combinations of products often bought together:

moonmagic_phone_001_003

We found that there were over 25 combinations of products that are often bought together. 

Incorporating these combinations into our experiment would significantly complicate it. Moreover, the role of an A/B testing is to validate one hypothesis at a time. Adding the carousel to the cart was our primary hypothesis, so we decided to use the already built-in recommendation engine on the site. There were two built-in recommendation engines functioning on the site. We created a regression model to identify which one of them drives more sales. We found that users who interacted with one of them converted x3.6 better than the site average. This was our winner. 

 

Hypothesis

We hypothesized that by introducing a cross-sell carousel into the cart we will increase the average number of items purchased per transaction and increase the AOV.

The risk that we were running with this hypothesis is that by distracting users from conversion in the cart and leading them to other PDPs, we may negatively impact the CR. However, even if this would be the case we expected the growth in AOV to compensate for the decrease in CR.

So our solution was to introduce a cross-sell carousel in the cart.

Control variation

moonmagic_phone_001_001

Alternative variation

moonmagic_phone_001_002

Validating the hypothesis

To validate this hypothesis we conducted an experiment that measured the impact of the cross-sell carousel in the cart on the CR, AOV and ultimately ARPU. 

 

Experiment details:

Experiment type: A/B test

Traffic split: 50:50

Key metric: ARPU

Targeting condition: We targeted the test only toward users who added a product to the cart and viewed the cart. This was done to minimize data noise and achieve statistical significance of the result quicker.

Number of users taking part in the experiment: 2 518 users

Number of conversions recorded during the experiment: 694 conversions

Result

The alternative variation that introduce a cross-sell carousel in the cart has shown a 9% uplift in ARPU, 4% uplift in CR and 9% uplift in AOV. The result had 95% statistical significance and the power of the experiment of 80%.

So users who experienced the alternative variation not only purchased more products per transaction elevating the AOV, but also were more likely to complete the purchase positively influencing the CR.

Learn how to grow your AOV with effective use of a recommendation engine.

Sign up to a free 20 min review of your site with a CRO strategist and get 3 CRO hypotheses for FREE.
Get 3 CRO hypothesis for FREE

Increase conversions performance-based

(pay only for actual % growth)

Start here
Leboutique Microsoft Сomodo Preply Prom