How to A/B test checkout upsells
To find which upsell works best in your checkout, you can now use A/B testing within UpsellPlus.
To enable A/B testing, go to the UpsellPlus app. Create an offer, choose ‘Checkout’
Once you’ve added a product and set up the design and discounts you want, you’ll see the following option.
Click the button! The app will copy your current offer and create a B version. You can toggle between the two versions with the tabs at the bottom.
Here’s an example of a simple A/B test setup that will test the callout banner that introduces the upsell offers in checkout: ‘Frequently bought together’ vs ‘73% of customers also bought’
You can also see the impressions, conversion and revenue for each version. Once you publish the offer to your checkout, 50% of shoppers will see version A and 50% of shoppers will see version B.
The smart rules are not part of the A/B version testing as that would invalidate the 50/50% visibility for shoppers.
Some tips to run an effective A/B test:
1. Set a goal and hypothesis
To get a useful result out of an A/B test, define your hypothesis. In simple words, say what you want to test:
“Changing the discount from 20% to 30% of my socks checkout upsells will increase the conversion rate from 3% to 5%”
“The CTA button text ‘ADD TO CART’ will convert 1% higher than ‘add to cart’”
“A stack layout for my multiproduct checkout upsell will earn 20% more revenue than a carousel layout”
This will help you understand what the goal was of the test when you come back to it. You can add this hypothesis as the internal name of the upsell offer at the very top of the page.
2. Only test one variable at a time
Choose 1 variable to change between the two versions and let the test run until you hit at least 30 conversions.
Ideas:
Discount %: 20% vs 30%
Callout banner text
Design
Button CTA text
Button color
Upsell product
Multiproduct layout: stack vs carousel
Upsell description
Upsell marketing message
3. Get a statistically significant result
The UpsellPlus app will declare a winner after a statistically significant result has been reached for your store. It will indicate a winner has been found with the ‘trending’ badge next to the winning version.
This badge will also show in your offer overview to notify you that a winner has been determined.
Note that finding a statistically meaningful result might take a while if you’re having a low conversion rate for the upsells. Give the A/B test enough time to produce useful data.
4. Create a 'control' and a 'challenger'
When you had an upsell run without an A/B test, keep that unchanged (‘control’) and add a ‘challenger’ with one variable that’s updated to test a hypothesis.
Something to note: when you choose to run an A/B test, you can no longer switch the offer from a manually chosen product to an automagically (AI) chosen product. This would invalidate the A/B test as too many variables would change.
To enable A/B testing, go to the UpsellPlus app. Create an offer, choose ‘Checkout’
Once you’ve added a product and set up the design and discounts you want, you’ll see the following option.
Click the button! The app will copy your current offer and create a B version. You can toggle between the two versions with the tabs at the bottom.
Here’s an example of a simple A/B test setup that will test the callout banner that introduces the upsell offers in checkout: ‘Frequently bought together’ vs ‘73% of customers also bought’
You can also see the impressions, conversion and revenue for each version. Once you publish the offer to your checkout, 50% of shoppers will see version A and 50% of shoppers will see version B.
The smart rules are not part of the A/B version testing as that would invalidate the 50/50% visibility for shoppers.
Some tips to run an effective A/B test:
1. Set a goal and hypothesis
To get a useful result out of an A/B test, define your hypothesis. In simple words, say what you want to test:
“Changing the discount from 20% to 30% of my socks checkout upsells will increase the conversion rate from 3% to 5%”
“The CTA button text ‘ADD TO CART’ will convert 1% higher than ‘add to cart’”
“A stack layout for my multiproduct checkout upsell will earn 20% more revenue than a carousel layout”
This will help you understand what the goal was of the test when you come back to it. You can add this hypothesis as the internal name of the upsell offer at the very top of the page.
2. Only test one variable at a time
Choose 1 variable to change between the two versions and let the test run until you hit at least 30 conversions.
Ideas:
Discount %: 20% vs 30%
Callout banner text
Design
Button CTA text
Button color
Upsell product
Multiproduct layout: stack vs carousel
Upsell description
Upsell marketing message
3. Get a statistically significant result
The UpsellPlus app will declare a winner after a statistically significant result has been reached for your store. It will indicate a winner has been found with the ‘trending’ badge next to the winning version.
This badge will also show in your offer overview to notify you that a winner has been determined.
Note that finding a statistically meaningful result might take a while if you’re having a low conversion rate for the upsells. Give the A/B test enough time to produce useful data.
4. Create a 'control' and a 'challenger'
When you had an upsell run without an A/B test, keep that unchanged (‘control’) and add a ‘challenger’ with one variable that’s updated to test a hypothesis.
Something to note: when you choose to run an A/B test, you can no longer switch the offer from a manually chosen product to an automagically (AI) chosen product. This would invalidate the A/B test as too many variables would change.
Updated on: 12/12/2022
Thank you!