Exclude users from your experiments with holdout groups

This article will help you:

• Understand how holdout groups work in Amplitude Experiment
• Create, manage, and analyze a holdout group and the experiments in it
• Delve deeper into holdout groups with use case examples

Sometimes it can be useful to keep a certain percentage of users from viewing an experiment. This is especially true when measuring the long-term, combined effects of multiple experiments, because statistical significance in one experiment may not reflect the true, cumulative impact of your experiments.

Amplitude Experiment lets you easily exclude users from your experiments by creating a holdout group and adding experiments to it. Holdout groups are especially useful for measuring the long term impact of your rolled-out variants, and measuring the lift of your experimentation program as a whole.

To learn more about the underlying implementation, see documentation in the Amplitude Developer Center.

Before you begin

When using holdout groups, there are a few things to keep in mind:

• Set the holdout percentage between 1% and 10%.
• Withholding a significant portion of your total traffic can extend the time your experiments need to reach a conclusion, especially if you’re not getting a high traffic volume already.
• Don’t add a running experiment to a holdout group. This can severely compromise the integrity of your data by unassigning users from the active experiments. Instead, add experiments to a holdout group before they’ve started running.
• Don’t remove a running experiment from a holdout group. This can compromise the integrity of your data by assigning users to the active experiments.
• Don’t delete a holdout group with running experiments, for the same reason. Instead, delete the holdout group after all experiments in the group have concluded.

Create a holdout group

To create a holdout group and add your experiments to it, follow these steps:

1. In Amplitude Experiment, navigate to the Experiment Groups tab in the Experiments page. (If you’re trying to do this from within an experiment, open the Configure tab and click See Holdout Groups.) Here, you can see all your holdout groups and be able to create new ones. 
2. To add a new holdout group to your project, click Add a new holdout group.

If you have existing groups, click Create A New Group, and then select Holdout Group in the drawer.

3. In the Holdout group settings modal, enter the name, description, and holdout percentage for the group. You can also view and change advanced settings such as the evaluation mode and bucketing key of your group.

NOTE: The holdout percentage cannot be changed once you’ve created the group. This ensures consistent bucketing, as well as a consistent user experience.

4. Add experiments to your holdout group.

5. Click Add Group to finish the process.

Manage holdout groups

Manage your holdout groups from the Experiment Groups tab or from within an experiment:

1. In the Experiment Groups tab, scroll down the table until you find the group you want to edit.
2. Click the edit icon.
3. Make your changes in the Holdout group settings modal and click Save.

Alternatively, if you are within an experiment that is part of a holdout group, follow these steps: 

1. Navigate to the Configure tab.
2. Click the name of the group you want to edit
3. Make your changes in the Holdout group settings modal and click Save.

Analyze a holdout group

Analyze your holdout groups using an Experiment Results chart.

To create a pre-populated Experiment Results chart, follow these steps: 

1. Navigate to the Experiments page and open the Experiment Groups tab.
2. Find the holdout group you want to analyze and click the chart icon.
3. Click Open in Analytics.

A new Experiment Results chart will open, with the following fields prefilled:

• • • Exposure event
    • Segments for holdout and on
    • Statistical method set to T-test (Samples per variant needed set to 10,000)
    • Analysis date range

From here, select the primary metric and start analyzing the impact of your holdout group.

Advanced use cases

• Adding an experiment to multiple holdout groups will limit an experiment's traffic even more. This is because each user must be evaluated for each holdout group they belong to.

For example, imagine these two holdout groups: 

• • Holdout group 1 contains experiment A and experiment B, with a holdout percentage of 5%
  • Holdout group 2 contains experiment A and experiment C, also with a holdout percentage of 5%

Experiment A will receive 0.95 * 0.95 = 0.9025, or 90.25% of the total traffic.

Instead of adding an experiment to multiple holdout groups, create a single group with all the relevant experiments instead. In the example above, that would be one group containing experiments A, B, and C.

• Adding an experiment to a holdout group and a mutual exclusion group will further limit the amount of traffic to the experiment, as each user will be evaluated for both the holdout group and the mutual exclusion group.

For example, imagine the following holdout group and mutual exclusion group: 

• • The holdout group has a holdout percentage of 5% and contains experiment A
  • The mutual exclusion group has half the traffic directed to experiment A in slot 1, with the other half going to experiment B in slot 2

Then experiment A receives 0.95 * 0.5 = 0.475 or 47.5% of the total traffic.

Learn more in this Amplitude Help Center article about mutual exclusion groups.