The experiment workflow: Allocate your experiment's traffic

  • Updated

This article will help you:

  • Set traffic percentages for each of your variants
  • Limit experiment exposure based on user segments

The next step of the experiment workflow is to determine how much traffic each variant receives, and to specify who will see each variant. You'll do this in the Configure tab's Allocation panel. Here you can define user segments that will see your experiment, specify the percentage of users who will be exposed to your experiment, and set the relative distribution weights of each variant.

Target individual users

Amplitude Experiment allows you to target your experiments to individual user or device IDs. This is mostly used for QA purposes. For more information, see our Help Center article on QAing your experiments before rollout.

Define user segments

Defining a user segment is useful if you’d like to limit experiment exposure to users in a specific geographical location, or those who belong to certain demographic groups, or those who meet certain usage thresholds in your product (i.e., power users).

To define a user segment, under Who should be eligible for this experiment?, select Targeted Users and click into Segment 1. Then follow the same steps you’d use to build a user segment in Amplitude Analytics:

  • To target your experiment based on cohorts, select Cohort and choose a cohort you've previously created in Amplitude Analytics. Cohort membership is evaluated and updated once an hour. 
  • To target based on user properties from Amplitude Analytics, select the user property you're interested in and enter the appropriate values. User properties come from the end user profile maintained by Amplitude Analytics, and are updated based on your analytics instrumentation. (If you need to evaluate user properties in real-time, add the value of the user property to the context you pass to the SDK or REST API.)
  • To target based on custom user properties (i.e., properties that don't exist in Amplitude), type the user property name into the dropdown and enter values to match on for the property. (Rules containing custom user properties can only evaluate against values explicitly defined in the context passed to the SDK or REST API, since Amplitude does not know about these properties otherwise.)

Add more segments by clicking + Add Segment.

Anyone not assigned a variant as part of a rule-based user segment—and who was not targeted individually—will be assigned a variant based on the configuration in the All other users section (discussed in Step 4 below).

All of Amplitude’s user properties and cohorts are available to use in defining user segments. Note that the value for any user property included in a rule-based user segment will be the most recent value received by Amplitude. There is no limit on the number of user segments you can include here. Any user who belongs to more than one segment included in an experiment will be assigned to the first one they match. 

NOTE: Caching expiration differs with the properties you use to define user segments. All user properties will always include the latest values your product has sent, defaulting to the most recent value in the event that Amplitude has not received your product’s data. Cohorts will be synced every hour.

Stratified sampling and experiment bias

In some rare cases, you may want to run an experiment in which you allocate traffic differently for the user segments you’ve included:

  • Segment 1: Country = USA || 80% treatment, 20% control
  • Segment 2: Country = Canada || 50% treatment, 50% control

However, this can introduce bias into your experiment results. For that reason, it is encouraged that you adhere to uniform allocation ratios across all user segments in an experiment. 

In many cases, non-uniform allocation ratios happen inadvertently, when users change their rollouts and variants while an experiment is running.

Amplitude Experiment gives you the option to use stratified sampling (i.e., non-uniform allocation ratios) if you need it. Just switch the Allow rollout controls per segment toggle to On. (This option will only be visible if you've selected Targeted Users instead of All Users.)

allow_rollout_controls.png

This switch is visible only for experiments, and not for feature flags. It is disabled while your experiment is active. 

Rollout percentage

Next, set the rollout percentage for this experiment. This is the percentage of the users included in the experiment’s user segments who will participate in the experiment.

experiment

After that, you’ll need to tell Amplitude Experiment how to distribute your variants. This will determine how many users who’ve been exposed to the experiment will see each variant. 

You can then set separate rules for everyone not covered by the segment you created in step 3. If, for example, you only want the experiment rolled out to the specific cohorts you targeted earlier, set the rollout percentage to zero in the All other users section.

Note that Amplitude Experiment evaluates for users included in rule-based user segments before those not covered by a user segment. However, it evaluates for any individual user or device IDs prior to both. For more information, see this article in the Amplitude Developer Center.

You can see the total effective distribution of variants across all users in the Summary section.

Allocate internal users to a variant

If you want to allocate your internal users to the treatment of an experiment, but you don’t want to skew your experiment data because the internal users are not representative of your customer base, you have a couple of options:

  1. If you are only including a few internal users, you can use the individual inclusions. However, this isn't scalable if you have a large number of users to allocate.

  2. You can make separate flags, one for internal users and one for your experiment. Then you can connect them via an OR statement when evaluating each user.

  3. Include internal users in the targeting for the experiment, but remove them when you run your analysis. To do this, navigate to the Analyze tab and click Open in Analytics. Add the condition that the email address does not contain "amplitude" to both experiment variants where you would filter on user segments.

  4. Finally, you can create a third variant, send all internal users to that variant, and then simply exclude it from your analysis. However, you may need to run your own Bonferroni correction because, for the purposes of your analysis, that third variant is not actually a legitimate variant.

Next, you'll roll out your experiment to your users.