The experiment design phase: Define your success metrics

To add metrics to your experiment, scroll to the Success Metrics panel. Here, you’ll tell Experiment what you want your primary metric to be, as well as any secondary metrics. The primary metric determines whether your hypothesis has been accepted or rejected—and therefore, whether your experiment has succeeded or failed.

There’s a lot riding on your primary metric, so it’s important to select the right one. If you’re not experienced in A/B testing, it can be hard to know which one that is. But if you know what to look for, your odds of a successful variant improve dramatically:

• • Try to identify the single user action that will tell you if your variant is successful.
  • Measure an event that is directly affected by the change you’ve made in your variant.
  • Pick an event that fully captures the user behavior you’re trying to affect.

One common mistake is defaulting to a revenue metric when it’s not appropriate. This happens when your variant introduces a change that is separate from the metric you’ve selected. If your variant changes how your product page looks and functions, you should choose a metric on that page as your primary metric, instead of a revenue metric that might not come into play for several more steps down the funnel. 

Amplitude Experiment lets you define multiple success metrics when running an experiment. Unlike a primary metric, secondary metrics aren’t required—but they are often helpful. They can not only improve the quality of your analysis, but help evaluate whether it’s even worthwhile to roll out your experiment at all.

Set your experiment's success metrics

To set up the success metrics for your experiment, follow these steps:

1. Choose the exposure event. This is the event that users will have to perform before joining the experiment. For example, if your experiment has to do with a new design for a drop-down navigational menu, you'll want to limit your experiment to users who actually interact with that menu. So your exposure event might be click_menu.
2. Select your primary metric from the Select Metric dropdown list, or create a new metric. 
3. Next to Direction, specify whether you’re expecting the success metric to increase or decrease; if you’re not sure, choose Any.
   
   
   
   
4. Repeat this process for any secondary metrics you want to include.

Use the design calculator to estimate the sample size you'll need to achieve significant results in your experiment, given your success metric settings. Adjust the confidence level, statistical power, minimum detectable effect, standard deviation, and test type, as needed. For more information on these and other Amplitude Experiment concepts, be sure to see our glossary of key experimentation terms. 

Create a new metric

To create a new metric, follow these steps:

1. From the Select Metric dropdown, click Create new Metric.
2. In the Create Metric modal, give your new metric a name and select its type. A metric can be one of four specific types: unique conversions, average event totals, sum of property value, or the average of property value.
   
   
   
   
3. Choose the metric event, which is the event that best represents that metric. Then click Create.

Edit an existing metric

Once created, your metrics can be edited later. Be careful, though: when you edit an existing metric, it will change for every experiment that uses it. 

To edit an existing metric, follow these steps:

1. Click the Select Metric dropdown and hover over the metric you wish to edit.
2. Click Edit Full Definition. The Edit Metric modal will open.
   
   
   
   
3. Make the changes you need and click Save Changes.

The next step is creating and adding your variants.