This article will help you:
The next step of the experiment workflow is to determine how much traffic each variant receives, and to specify who will see each variant. You'll do this in the Configure tab's Allocation panel. Here you can define user segments that will see your experiment, specify the percentage of users who will be exposed to your experiment, and set the relative distribution weights of each variant.
Target individual users
Amplitude Experiment allows you to target your experiments to individual user or device IDs. This is mostly used for QA purposes. For more information, see our Help Center article on QAing your experiments before rollout.
Stratified sampling and experiment bias
In some rare cases, you may want to run an experiment in which you allocate traffic differently for the user segments you’ve included:
- Segment 1: Country = USA || 80% treatment, 20% control
- Segment 2: Country = Canada || 50% treatment, 50% control
However, this can introduce bias into your experiment results. For that reason, it is encouraged that you adhere to uniform allocation ratios across all user segments in an experiment.
In many cases, non-uniform allocation ratios happen inadvertently, when users change their rollouts and variants while an experiment is running.
Amplitude Experiment gives you the option to use stratified sampling (i.e., non-uniform allocation ratios) if you need it. Under Rule Based Segments, find the Use stratified sampling toggle switch and set it to On.
This switch is visible only for experiments, and not for feature flags. It is disabled while your experiment is active.
Define user segments
Defining a user segment is useful if you’d like to limit experiment exposure to users in a specific geographical location, or those who belong to certain demographic groups, or those who meet certain usage thresholds in your product (i.e., power users).
To define a user segment, scroll down to the Rule Based User Segments section and click into Segment 1. Then follow the same steps you’d use to build a user segment in Amplitude Analytics:
- To target your experiment based on cohorts, select Cohort and choose a cohort you've previously created in Amplitude Analytics. Cohort membership is evaluated and updated once an hour.
- To target based on user properties from Amplitude Analytics, select the user property you're interested in and enter the appropriate values. User properties come from the end user profile maintained by Amplitude Analytics, and are updated based on your analytics instrumentation. (If you need to evaluate user properties in real-time, add the value of the user property to the context you pass to the SDK or REST API.)
- To target based on custom user properties (i.e., properties that don't exist in Amplitude), type the user property name into the dropdown and enter values to match on for the property. (Rules containing custom user properties can only evaluate against values explicitly defined in the context passed to the SDK or REST API, since Amplitude does not know about these properties otherwise.)
Anyone not assigned a variant as part of a rule-based user segment—and who was not targeted individually—will be assigned a variant based on the configuration in the All Non-Targeted Users section (discussed in Step 4 below).
All of Amplitude’s user properties and cohorts are available to use in defining user segments. Note that the value for any user property included in a rule-based user segment will be the most recent value received by Amplitude. There is no limit on the number of user segments you can include here. Any user who belongs to more than one segment included in an experiment will be assigned to the first one they match.
NOTE: Caching expiration differs with the properties you use to define user segments. All user properties will always include the latest values your product has sent, defaulting to the most recent value in the event that Amplitude has not received your product’s data. Cohorts will be synced every hour.
Next, set the rollout percentage for this experiment. This is the percentage of the users included in the experiment’s user segments who will participate in the experiment. Choose one of the preset values, or manually enter your desired value.
After that, you’ll need to tell Amplitude Experiment how to distribute your variants. This will determine how many users who’ve been exposed to the experiment will see each variant. The weights are relative values: for example, if you give variant A a weight of 1 and variant B a weight of 4, then four times as many users will see variant B than variant A. (Bucketing is based on the bucketing key you select.)
You can then set separate rules for everyone not covered by the segment you created in step 3. If, for example, you only want the experiment rolled out to the specific cohorts you targeted earlier, set the rollout percentage to zero in the All Non-Targeted Users section.
Note that Amplitude Experiment evaluates for users included in rule-based user segments before those not covered by a user segment. However, it evaluates for any individual user or device IDs prior to both. For more information, see this article in the Amplitude Developer Center.
You can see the total effective distribution of variants across all users in the Summary section.
Next, you'll roll out your experiment to your users.