> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://help.upsellplus.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# How to measure the real revenue impact of your upsells with holdout experiments

**A holdout experiment proves how much extra revenue your upsells really add.** UpsellPlus hides offers from a small share of shoppers (the *holdout*) and compares them with everyone else (the *treatment*). The difference in conversion rate, average order value and revenue per session is the revenue your offers add, not just sales they take credit for.

- **Where:** Experiments → Holdout Experiments → **New Experiment**
- **Two scopes:** hold out **All offers** (UpsellPlus as a whole) or **Selected offers** (only the offers you pick)
- **Recommended holdout:** 10%


![Holdout experiment on selected offers with conversion, AOV and revenue per session lift](https://storage.crisp.chat/users/helpdesk/website/-/a/b/f/a/abfa0cd47f75e80/holdout-scope-choice_1yv80wu.png)

## Holdout or A/B test?

- **A/B test:** compares two versions of an offer to find the better one.
- **Holdout:** compares showing offers with not showing them, to measure the offers' total value.

## Create a holdout experiment

1. Go to **Experiments**, open **Holdout Experiments** and click **New Experiment**.
2. Enter an **Experiment Name**, for example *Q4 2026 revenue impact test*.
3. Under **Offers to hold out**, choose:
   - **All offers**: held-out shoppers see no UpsellPlus offers
   - **Selected offers**: held-out shoppers don't see the offers you pick and get your next offer instead. Pick at least one offer.
4. Choose a **Holdout Percentage**:

   | Holdout | Trade-off |
   |---|---|
   | 5% | Minimal impact on revenue, slower to reach significance |
   | **10%** | **Recommended:** good balance of speed and impact |
   | 15% | Faster results, more revenue held back during the test |
   | 20% | Very fast results, significant revenue held back |
   | 50% | Fastest results, half of revenue held back |
   | Custom | Any whole number from 1% |

5. Click **Create Experiment**, then **Start Experiment**.

Only one holdout experiment can run at a time. You can create drafts while another is running, but you have to stop the running one before starting a new one.

## Read the results

![Holdout report comparing treatment and holdout groups for two selected offers](https://storage.crisp.chat/users/helpdesk/website/-/a/b/f/a/abfa0cd47f75e80/holdout-selected-offers-lift_1u9gbol.png)

The experiment page shows:

- **Treatment vs Holdout:** sessions, orders, conversion rate, AOV and revenue for each group
- **Incremental Lift:** **Conversion Lift**, **AOV Lift** and **Revenue / Session Lift**, each marked **Statistically significant** or **Not significant** with its p-value

Until the result is significant (p ≤ 0.05), the page tells you to collect more sessions before drawing conclusions. Stats can lag by up to an hour; use **Refresh Stats** to update.

## Tips

- **Test one offer that "looks" great:** use **Selected offers** to check whether an offer with a high conversion rate really adds revenue, or mainly takes credit for products shoppers would have bought anyway.
- **Let it run:** stop only once the lift you care about is statistically significant.
- **Post-purchase offers** aren't held out by a Selected offers experiment.

## FAQ

**Will a holdout cost me revenue?**
A little, during the test: held-out shoppers don't see your offers. That's why 10% is recommended.

**Can I run a holdout and A/B tests at the same time?**
Yes. Only one holdout experiment can run at a time.

**Which plan includes holdout experiments?**
Holdouts are included on the Pro plan, along with A/B experiments.

**Why does it say "Not significant"?**
There aren't enough sessions yet to be confident the difference is real. Keep the experiment running.

## Related articles

- [How to run an A/B test](https://help.upsellplus.com/en/article/how-to-run-an-ab-test-on-an-upsell-offer-1se7j0/)
- [What can I A/B test?](https://help.upsellplus.com/en/article/how-to-run-an-ab-test-on-an-upsell-offer-1se7j0/)
- [How is upsell revenue calculated?](https://help.upsellplus.com/en/article/how-is-upsell-revenue-calculated-pat4az/)