This article will help you:
The decisions you make in the design phase set the stage for your experiment’s success. By putting more thought into your experiment’s purpose and goals before you start, you’ll be far more likely to glean useful, actionable insights from it.
To create a new experiment, first ensure you’ve created an environment and installed the SDK. Then follow these steps:
- Open Amplitude Experiment and click + New. From the drop-down menu, select Experiment.
- In the Create Experiment modal, choose the project that will house this experiment from the Projects drop-down menu.
- Amplitude Experiment uses flags to include experiments within your product. In the appropriate fields, give your flag a name and a description. Experiment will automatically generate the flag key for your experiment from the name you choose; this key will act as an identifier for the flag used in your codebase.
- When you’re done, click Create. Experiment will open a blank template for your flag.
- The first step is adding context to your experiment. The Context panel is an Amplitude notebook template that gives you a space to explain the objectives, target audiences, and goals of the experiment for stakeholders and other members of your team. This means that anytime another person views your experiment, they’ll be able to understand it at a glance.
To fill in the Context panel, click Add Context. Click into a field to edit it.
NOTE: Filling out the Context panel is not required to run an experiment, but it’s strongly recommended.
- Next, scroll to the Metrics panel. Here, you’ll tell Experiment what you want your success metric to be. The success metric determines whether your hypothesis has been accepted or rejected—and therefore, whether your experiment has succeeded or failed.
Start with setting the metric type: are you interested in unique conversions, average event totals, or the average sum of a property value? Next, click Select event… and choose the event you want to use as your success metric. Finally, next to Expected Change, specify whether you’re expecting the success metric to increase or decrease; if you’re not sure, choose Any.
Use the design calculator to estimate the sample size you'll need to achieve significant results in your experiment, given your success metric settings. Adjust the confidence level, statistical power, minimum detectable effect, standard deviation, and test type, as needed. For more information on these and other Amplitude Experiment concepts, be sure to see our glossary of key experimentation terms.
There’s a lot riding on your success metric, so it’s important to select the right one. If you’re not experienced in A/B testing, it can be hard to know which one that is. But if you know what to look for, your odds of a successful variant improve dramatically:
- Try to identify the single user action that will tell you if your variant is successful.
- Measure an event that is directly affected by the change you’ve made in your variant.
- Pick an event that fully captures the user behavior you’re trying to affect.
One common mistake is defaulting to a revenue metric when it’s not appropriate. This happens when your variant introduces a change that is separate from the metric you’ve selected. If your variant changes how your product page looks and functions, you should choose a metric on that page as your success metric, instead of a revenue metric that might not come into play for several more steps down the funnel.
- The final step in designing your experiment is to create at least one variant. Experiment will compare your variants with the control, which is usually your product’s current user experience. (This way, Experiment measures the performance of the variant against a known quantity, the performance of your app as it is today.)
To add a variant to your experiment, click + Create Variant. The Create Variant modal will appear.
Type in a name and a description for your variant in the appropriate fields. Experiment will automatically generate the variant value from the name you enter. The variant value is a string that you’ll use as a flag in your codebase. When you’re done, click Create Variant.
There is no limit to the number of variants you can add to an experiment, but adding too many can make it harder for your experiment to reach statistical significance. Try to keep your experiments limited to a handful of variants, at most.
- You can also add a payload to your variants. A payload is simply extra data that can dynamically change a variant’s experience without requiring you to write more code.
For example, imagine you’re testing a new splash screen on a marketing webpage. You might get early results that suggest different content might improve the performance of the splash. Instead of going into your codebase and making changes to the variant there, you can just include those changes in a payload, and Experiment will implement them automatically.
To add a payload to a variant, click next to Payload in the variant’s panel. The Edit Payload modal appears. Paste or type your code into the window and click Apply to set the payload for this variant.
- Be sure to save your changes regularly.
Now that you’ve finished designing your experiment, the next step is to roll it out to your users.