FAQ: Experiment Results

  • Updated

This article covers frequently-asked questions about Amplitude's Experiment Results chart.

Why is my graph displaying an error state?

A common mistake is to attempt to generate a chart using only one variant.

The Experiment Results chart needs something to compare your control to in order to generate results. If you neglect to include both the control and at least one variant, your chart will not display anything.

Why is reaching significance taking longer than it should?

When using a T-test, be sure to plug in the number of samples per variant before launching the experiment. You can get this information from the duration estimator. With a T-test, you will have to wait until your experiment reaches the specified sample size before Experiment Results will run the p-value and confidence interval computations.

With sequential testing, even with a large MDE, it can take some time to reach statistical significance if your experiment’s lift is small. A T-test will generally require fewer samples to detect the same lift.

 

How is the Retention metric calculated?

Amplitude uses two parameters to calculate Return On for the Retention metric:

  • The return event: The event you hope the user performs after the exposure event (aka, the starting event). The user is counted as retained if they trigger the return event.
  • Return on the nth day: The number of days you want to see between a user performing the exposure event and the return event. This parameter is calculated in 24 hour increments and does not use calendar dates. 

For example, if we have an exposure event T1 and a return event T2, with a return on the nth day value of two (two days, or 48 hours), the user will be counted as retained if they perform the return event anytime between two days (48 hours) and two days plus 24 hours (72 hours).