Articles on: Experiments

How to measure the real revenue impact of your upsells with holdout experiments

A holdout experiment proves how much extra revenue your upsells really add. UpsellPlus hides offers from a small share of shoppers (the holdout) and compares them with everyone else (the treatment). The difference in conversion rate, average order value and revenue per session is the revenue your offers add, not just sales they take credit for.

Where: Experiments → Holdout Experiments → New Experiment
Two scopes: hold out All offers (UpsellPlus as a whole) or Selected offers (only the offers you pick)
Recommended holdout: 10%

Holdout experiment on selected offers with conversion, AOV and revenue per session lift

Holdout or A/B test?

A/B test: compares two versions of an offer to find the better one.
Holdout: compares showing offers with not showing them, to measure the offers' total value.

Create a holdout experiment

Go to Experiments, open Holdout Experiments and click New Experiment.
Enter an Experiment Name, for example Q4 2026 revenue impact test.
Under Offers to hold out, choose:
- All offers: held-out shoppers see no UpsellPlus offers
- Selected offers: held-out shoppers don't see the offers you pick and get your next offer instead. Pick at least one offer.

Choose a Holdout Percentage:

Holdout	Trade-off
5%	Minimal impact on revenue, slower to reach significance
10%	Recommended: good balance of speed and impact
15%	Faster results, more revenue held back during the test
20%	Very fast results, significant revenue held back
50%	Fastest results, half of revenue held back
Custom	Any whole number from 1%

Click Create Experiment, then Start Experiment.

Only one holdout experiment can run at a time. You can create drafts while another is running, but you have to stop the running one before starting a new one.

Read the results

Holdout report comparing treatment and holdout groups for two selected offers

The experiment page shows:

Treatment vs Holdout: sessions, orders, conversion rate, AOV and revenue for each group
Incremental Lift: Conversion Lift, AOV Lift and Revenue / Session Lift, each marked Statistically significant or Not significant with its p-value

Until the result is significant (p ≤ 0.05), the page tells you to collect more sessions before drawing conclusions. Stats can lag by up to an hour; use Refresh Stats to update.

Tips

Test one offer that "looks" great: use Selected offers to check whether an offer with a high conversion rate really adds revenue, or mainly takes credit for products shoppers would have bought anyway.
Let it run: stop only once the lift you care about is statistically significant.
Post-purchase offers aren't held out by a Selected offers experiment.

FAQ

Will a holdout cost me revenue?
A little, during the test: held-out shoppers don't see your offers. That's why 10% is recommended.

Can I run a holdout and A/B tests at the same time?
Yes. Only one holdout experiment can run at a time.

Which plan includes holdout experiments?
Holdouts are included on the Pro plan, along with A/B experiments.

Why does it say "Not significant"?
There aren't enough sessions yet to be confident the difference is real. Keep the experiment running.

Updated on: 14/10/2026

Was this article helpful?

Thank you!