This doc will teach you how to:
- Instrument your app for A/B testing (split testing).
- Review the results of A/B tests in Amplitude.
A/B Testing is a method of conducting controlled, randomized experiments with the goal of improving a website or application metric. With our AB Test View (available for Growth and Enterprise plans), Amplitude enables you to measure the impact of your experiments by seeing how each experiment group behaves in your application.
Table of Contents
- Prerequisites: Instrument Your Experiments
- How Many User Properties Should I Send?
- Viewing the Results in Amplitude
Prerequisites: Instrument Your Experiments
The first step to reviewing the results of your experiments is instrumentation. We recommend using user properties to associate a user with a given experiment variation. User properties reflect traits about the individual person using your app. User properties can be used to segment your analysis within the Amplitude platform (see the section below for an example).
There are two main ways to update a user property:
1) SDKs & HTTP API: Update user properties on event action.
- How: User properties can be sent with each event via our SDKs or HTTP API.
- Pros: User properties will take effect at the moment the event is sent and exist with the user for all subsequent events until the property values are explicitly updated.
- Cons: These events will count towards your monthly event volume. Further, users will be counted as 'active users' by default by these events, so you will need to ensure any A/B testing-related events are marked as inactive events.
2) Identify API: Update user properties without sending an event.
- How: Our Identify API allows you to update a user property without having to send an event.
- Pros: Can asynchronously update a user property without sending an event and will not impact your monthly event volume count.
- Cons: The user property will not take effect until the user performs an action. This usually is not an issue for most experiments, but it may have an impact on experiments that are aimed to track whether or not inactive users are returning to your application.
For example, suppose you are testing the effectiveness of an email to get users who have been inactive for more than 7 days to return to your app. If the Identify API is used to update a user property, the user property will only be applied for those users who have returned to perform any event in your application. If a user remains inactive, even after receiving the email, the user property will not be applied to this user. As a result, this inactive user will not be included in the experiment group that has received the email because the user property was not attached to the user. For these kinds of use cases, we recommend option #1 to represent experiments that reach the user (eg. an event called "Email Sent"). More info on how user properties are synced here.
How Many User Properties Should I Send?
We have seen two approaches for instrumenting split tests within Amplitude:
1) Use one user property per experiment.
- All user properties are received as key-value pairs. This approach sets the experiment name as the key and all variations of the experiments as the potential values.
- User Property Key: "Experiment 1"
- User Property Value: "variation_a"
- Pros: Can easily select experiments to segment by from the user segmentation tab.
- Cons: Can result in an overwhelming list of user properties based on the number of experiments being run.
2) Use one user property for all experiments.
- All user properties are received as key-value pairs. This approach sets the key to 'Split Tests' (or something similar) and stores the values in an array.
- User Property: "Split Tests"
- User Property Value: ["experiment_1_value", "experiment_2_value"]
- You can segment on the user property 'Split Tests' by selecting the appropriate value or test group in right module of the chart control panel.
- Pros: You will only have one user property related to your split testing (rather than one per experiment), so your user property list will be more manageable in the dashboard.
- Cons: Arrays are limited to 10,000 characters if append or prepend is used. If an array were to exceed the character limit, then the characters past the threshold will not be recorded.
In addition to instrumenting split test experiments on your own, we have a full integration with Optimizely that will automatically update user properties for each experiment. Please read the integration guide for more details.
Viewing the Results in Amplitude
You can begin reviewing the results of your split tests after user properties have been updated for each experiment group. The AB Test View (available for Growth and Enterprise plans) is a feature within Funnel Analysis that will allow you to perform this analysis.
You can compare the activity between experiment groups by using the right module of the chart control panel. To segment by your experimental groups in Amplitude, simply use the Segmentation sidebar to add your experimental groups.
You can also view the impact of an experiment on in Retention and Event Segmentation charts.