Articles on: Experiments

How to run an A/B test on an upsell offer

An A/B test shows different versions of the same offer to different shoppers and tells you which one earns more. Pick an offer, choose what to change (for example the button text, layout or discount), split the traffic, and UpsellPlus tracks each version until there's a clear winner. Add more variants to run a multivariate test.


  • Where: Experiments → Create Experiment
  • Up to 5 variants, including the original (control)
  • To measure whether offers add revenue at all: use a holdout experiment instead



A/B test setup testing the call to action, with results showing the winning variant


What can I test?


For each variant, select the fields to change:


  • Banner Text and Banner Subtext
  • Call to Action
  • Layout
  • CTA Styles and Card Styles
  • Learn More Link
  • Discount
  • Products


Create an A/B test


Step 1: Setup

  1. Go to Experiments and click Create Experiment.
  2. Enter an Experiment Name and a Hypothesis, for example Changing the CTA to "Buy Now" will increase conversion rate by 15%.
  3. Choose the Base Offer.
  4. Choose a Primary Metric: Conversion Rate, Revenue Per User, Add to Cart Rate or Average Order Value. Optionally add Secondary Metrics to monitor.


Step 2: Variants

  1. Control (Original) is the offer as it is.
  2. For each variant, give it a name, select the fields to test and enter the new values. The preview shows each version on desktop and mobile.
  3. Click + Add Variant for a multivariate test (up to 5 variants in total).


Step 3: Traffic & Duration

  1. Choose Equal Split or Custom traffic allocation. Each variant needs at least 5% of traffic, and the total must be 100%.
  2. Check the estimate: enter your Baseline Conversion Rate, Minimum Detectable Effect and Estimated Daily Traffic to see the Required Sample Size and Estimated Duration.
  3. Choose Start immediately or Schedule for later, and optionally Auto-stop after a number of days.
  4. Click Create & Start (or Create as Draft).


Auto-end and guardrails


Turn on Auto-end to stop the test when the result is statistically significant:


  • Confidence threshold: 90%, 95% or 99%
  • Minimum runtime: the test won't end before this many days


The winning variant is declared, but not applied automatically: you review the results and apply it. Guardrail metrics set to Stop test always apply, even with auto-end off.


Read and conclude the test


A test ends with one of these outcomes: Winner found, No clear winner / Inconclusive, or Manual stop. Tests can also end because a guardrail was violated, there was no meaningful difference after twice the required sample size, or the scheduled end date was reached.


While a test is running, you can't change the offer's placements.


Tips


  • Test one big idea at a time (for example discount vs no discount) before small styling changes.
  • More variants need more traffic. If the estimate is over 60 days, test fewer variants or a bigger change.
  • Home suggests offers that are ready for a test (verdict Test).


FAQ


What's the difference between an A/B test and a holdout?
An A/B test compares versions of an offer. A holdout compares showing offers with not showing them. See How to measure the real revenue impact of your upsells with holdout experiments.


Can I test showing an offer vs not showing it?
Yes, with a holdout experiment on Selected offers.


Which plan includes experiments?
A/B experiments and holdouts are included on the Pro plan.



Updated on: 14/10/2026

Was this article helpful?

Share your feedback

Cancel

Thank you!