This article will help you:
Previously, Amplitude customers had to use Amplitude Experiment’s feature flagging system if they wanted to take advantage of experimentation features built into the platform. With Experiment Results, Amplitude Analytics customers who have invested in a non-Amplitude feature flagging platform, whether third party or homegrown, can now take advantage of Amplitude’s planning, tracking, and analysis tools for Experiment—while still using the A/B tracking data generated by their own feature flagging platform.
Before you begin
Prior to using Experiment Results, you’ll need to ensure you’ve instrumented the metric events that are relevant to your experiment. Without them, you’ll be unable to create the success metrics and goals that Experiment Results needs to compare each variant in its analysis.
You will also want to make sure you’ve instrumented the necessary exposure events, which represent the delivery of a variant to a user participating in the experiment. See the Amplitude developer documentation on exposure tracking for more detailed information.
Analyze an A/B test using Experiment Results
To create an A/B test and see the results, follow these steps:
- Click Create New > Analysis > Experiment Results.
- In the Events module, click Define Metric to begin setting up your primary metric.
- In the Define Metric fly-out panel, use the drop-down menu to specify the metric type. There are five options:
- Unique conversions
- Event totals
- Sum of property value
- Average of property value
- Funnel Conversion
The first four are available for individual event metric analyses, while funnel conversion allows you to define a multi-step journey that must be completed in order for conversion to be counted.
Next, specify the event that will be used for this metric. You can also filter the event using a + where clause. When you’re finished, click Done.
- Optionally, click Add secondary metric to add a second, subordinate metric to the analysis. You can add multiple secondary metrics as necessary.
- Click Select event… to define your experiment’s exposure event. The exposure event is the event users must trigger to become part of the experiment.
In the Segmentation module, add your variants. All experiments require at least one variant, which is known as the control. You can use a previously-saved segment by clicking Saved Segments and selecting the one you want from the list. You can also create a new variant by clicking Add Experiment Variant.
Choose the properties and values that will define your variant and click Apply.
- Click Add experiment variant to add more variants as necessary to reflect the experiment setup in your feature flagging system.
Amplitude will calculate your statistical results on the fly and display them in the stats table, located in the Metrics Module.
Interpret your results
While the specifics may vary depending on the metric types you’re using, you’ll see three charts depicting your results:
- Absolute values by variant
- Absolute values with confidence interval over time
- Cumulative exposure
Experiment Results FAQs
Why is my graph displaying an error state?
A common mistake is to attempt to generate a chart using only one variant.
The Experiment Results chart needs something to compare your control to in order to generate results. If you neglect to include both the control and at least one variant, your chart will not display anything.
Why is reaching significance taking longer than it should?
When using a t test, be sure to plug in the number of samples per variant before launching the experiment. You can get this information from the sample size calculator. With a t test, you will have to wait until your experiment reaches the specified sample size before Experiment Results will run the p value and confidence interval computations.
With sequential testing, even with a large MDE, it can take some time to reach statistical significance if your experiment’s lift is small. A t test will generally require fewer samples to detect the same lift.