What You Will Learn in This Article
This article represents Part Two our our Funnel Analysis Help Centre articles. This article represents an in-depth guide to each metric available in Funnel Analysis.
In this article you will learn how each order of computation works, how each metric calculates conversion (for example: how many times a user can enter a funnel), what a conversion window is and how it influences the percentage of converted users. This article also covers the math behind some of our more advanced metrics.
Use these features to measure and test hypothesis regarding user conversion, conversion over time or see what is the statistical impact of your A/B tests.
Before you start reading this article, we recommend that you have a look at the prerequisites list below. The information presented in this list will help you out with both useful information and good to know practices but will also contain links to related topics and Help Centre articles:
- This article represents part two out of three articles regarding Funnels. Please see the link to Funnel Analysis - Getting Started here and the link to Funnel Analysis - Conversion Drivers here;
- A/B Test View and A/B etc is a feature that is only available to the Growth and Enterprise plans;
- Please note that the user conversion for each metric is calculated differently and results are not expected to match;
- Please feel free to have a look at the other two articles on this topic: Funnels Analysis - Getting Started article and the Funnels Analysis - Conversion Drivers.
Table of Contents
- Acting on Funnel Results
- Computation Methods
- This Order
- Repeat Events in Your Funnel Definition
- Any Order
- Exact Order
- Simultaneous Events
- User Property Segmentation
- Session Based Funnels
- Conversion Over Time
- Time to Convert
- Conversion Drivers
- A/B Test View (Growth and Enterprise)
- A/B Test Improvement
- A/B Test Significance
- Statistical Significance
- Breakdown Table
- Conversion by Event & User Property
- Video Walkthrough
Acting on Funnel Results
You can click on any segment of a funnel to see detailed information about users within the selected segment. See Microscope for more information.
- Show User Paths: View the event path users took within your funnel segment.
- View Users: See a list of User ID's that reached that step in the funnel.
- Investigate Conversion Drivers: See Conversion Drivers section of this article.
- Create Cohort (Enterprise): Enterprise customers can create a Behavioral Cohort from the users in a selected funnel segment. Once a Cohort is created, you can further analyze user behavior and retention on your new cohort.
- Download Users (Enterprise): You can download a CSV file of all users and their information in a selected funnel segment. This feature is only available on our Enterprise Plan.
There are three ways to compute funnels: "this order" (ordered), "any order" (unordered), and "exact order" (no events in between). The default funnel computation mode is "this order."
Events must be performed in the order they are set for the user to be converted. For example, a user cannot perform:
Event C -> Event B -> Event A
and be converted. Event B must follow Event A before Event C, the user must have done the steps in the following order to be converted:
Event A -> Event B -> Event C
Note that if the user performs:
Event A -> Event C -> Event B -> Event C
then the user would be counted as converted because they eventually performed the steps in order.
Repeat Events in Your Funnel Definition
In "this order" you can have repeated events in the funnel definition. For example, if your funnel definition is:
Event A -> Event A -> Event B
a user is counted as converted if he did Event A at least two times and then did Event B later. There are no limits to the number of times an event can appear in the funnel definition. Repeated events are not supported in "any order" Mode.
Events can be performed in any order (up to the step the user drops off), and the user will be converted. Users cannot skip steps in Funnel charts. If a user skips the 2nd step in a 3-step any order funnel, this user will be considered as dropped off at the 2nd step even if the 3rd step was completed.
For example, suppose we have the following funnel step setup:
A user could have done the steps in the following order:
Event 1 -> Event C -> Event B
and still be counted as a converted user (assuming they perform all steps within the conversion window). Users cannot skip steps in funnel charts so it’s any order up until the step they drop off.
A user will only be included in the funnel if the user has fired the event in the first step, Event 1. For example, if a user fired only Event B, the user would not be in the funnel. In order for the user to be in the funnel, the user must have performed the first event. So, the user must have fired Event 1 in order to "enter the funnel" (though the 2nd or 3rd step events could've been completed in any order: Event B -> Event C or Event C -> Event B).
And if a user completes Event B -> Event C -> Event 1, the funnel will count this user as converted only if all three events occurred during your selected time range or within your conversion window following the time range. If the user only completed Event 1 within your selected time range, the user will be counted as entered the funnel but dropped off. This is because the funnel chart does not account for events that occurred prior to your selected time range.
Events must be performed in the order they are set, with no events occurring in between, for the user to be converted. Say for example we have an exact order funnel defined as:
Event A -> Event B
In order to be counted as converted, a User cannot have done any events between Event A and Event B. So if a User performs Event A, but then a series of events are triggered in the background before the User performs Event B, they will not be counted as converted. If an event that is not part of the funnel definition is simultaneously fired in the same second, the User will still be counted as converted. This is because a second is the lowest time resolution we currently support.
Events that are marked “Hidden from Dropdowns” in the “Advanced” section of “Manage Data” section are ignored from the query. Note that depending upon your instrumentation settings you may see a difference in exact order funnels results and a user’s individual event stream or results from Pathfinder queries.
We round all time to seconds and so we maintain a one-second window to account for "simultaneous events". As a result, if there is any ambiguity with two different events, we will count both directions as a conversion. For example, if a user triggers Event B first, and then triggers Event A within one second, a funnel will count this as a conversion from:
Event A -> Event B or Event B -> Event A
If two of the same event types are sent within the same second, then we will count only one of those events.
User Property Segmentation
If you were to segment the data on a user property (using the "where" clause under All Users), the segmentation would be done based on the first step of the funnel. In the example above, this would be Event A. For example, suppose a user did:
Event B with the user property '[Amplitude] Country' = 'Canada',
then Event A with the user property '[Amplitude] Country' = 'United States'
If you were to do "...by Active country(s)" above All Users, it will show the above user in the '[Amplitude] Country' = 'United States' segment in the Event A step since the segmentation is done based on the first step of the funnel.
By utilizing the "holding constant" feature in the bottom module, you can build session-based funnels. To do this, hold constant '[Amplitude] Session ID':
A user must complete each step in the funnel with the same Session ID in order to be converted. Like all other properties that can be selected with the "holding constant" feature, a Funnel Analysis chart with this setup will no longer show unique users since users can complete the funnel multiple times, each time in a different session.
Conversion Over Time
Below the bar chart module is a line graph that shows the funnel conversion rates over time. The conversion rate graph shows conversion rates for users who entered the funnel on a specific date. So if a user enters a funnel on January 1st and later converts in the funnel on January 5th, they will be counted in the bucket for January 1st since that is the day they entered the funnel.
The percentages seen here are conversions per unique user per day/week/month. For instance, if a user enters the funnel (performs the first step of the funnel) on July 1st and July 2nd, and completes the funnel on July 30th (within 30 days of both July 1st and 2nd), then that user will be counted in both the conversion percentages for July 1st and 2nd.
In addition to looking at the overall conversion rate, this graph can display the conversion rate between funnel steps.
Time to Convert
Time to Convert helps you understand how long your users take to complete sequences or sets of important behaviors. It allows you to observe your users’ distribution of time between the different steps of your funnel in a histogram.
The percentages on the vertical axis represent the part-to-whole ratio of the users who converted within the given interval, relative to the number of all users who converted within the selected range.
As an example, let’s say that 100 users converted within a 15 minute window in the last 30 days. All 100 users had a conversion time of exactly 15 minutes. This means that 100% of users who converted within 15 minutes in the last 30 days will fall in the 15 minute bin.
The intervals on the horizontal axis are divided into 25 buckets. We automatically pick a bin size from 1 second, 10 seconds, 1 minute, 10 minutes, 1 hour, 1 day, 10 days, and 30 days where we have the most standard deviation in bins. The maximum and minimum time to convert is dependent on the conversion window and time range you’ve chosen.
If your analysis require a bin size that is not in line with the Amplitude default ones, you also have the option to create custom bins:
This feature can also help you understand how long your users take to complete specific steps of your funnel within your conversion window. If you click on “the entire funnel,” you will see a dropdown with pairs of event steps from your funnel.
Frequency displays a distribution of the number of times a user performs Event A before Event B is performed for the very first time in a given time interval. You can select the specific events you want to analyze below the conversion window. See below screen shot for reference.
For example, in the below example we see that 74.3% of users who reach Step 3 perform the ‘Download Song or Video’ event one time before purchasing a song or video completed within a 1 day time period.
Within each Funnels chart, when set to measure "conversion" using "this order", you can click into any of the steps after the initial event to open up the Conversion Drivers option, which allows you to look at the actions performed by users between steps in the funnel. This helps you clearly identify potential drivers of conversion, or drop-off.
AB Test View (Growth & Enterprise)
You are able to track user conversion with A/B test results in the default funnel view. However, with our advanced AB Test View, you can also receive a report on the statistical significance of each test variant to help better analyze the results and choose winners. To do this, first create the funnel you have instrumented the A/B test for. Then, create segments of users. The first segment you have selected will be the default baseline but you have the option to change this later in the bottom module of the chart control panel. You can change to the AB Test View by clicking the button in the bottom module. For example, let's say you have instrumented an A/B test to see if adding a picture to a song will increase the number of song purchases. We've instrumented the AB test under the 'A/B_Tests_Song_Purchase' user property and have three segments representing the baseline and the two variants. In the bottom module, you can choose to switch which segment you would like to use as the baseline.
A/B Test - Improvement
When the bottom module is set to "AB Test - Improvement", the chart will display the conversion rate for each segment over all steps. For example, we can see that the variant 'Purchase_With_Picture' is performing 6% worse than the baseline while the variant 'Purchase_No_Picture' is performing 3.75% worse than the baseline. Note we are using demo data, but with real segments that impact funnel conversion you should expect the better suited segment to positively impact your baseline, while a worse suited segment would negatively impact your baseline.
% Improvement over Baseline - Calculation Breakdown:
Improvement over baseline is the ratio of the mean variant (A) over the mean baseline (B), .
A/B Test - Significance
If you set the dropdown in the bottom module to "AB Test - Significance", then the chart will display the chance to outperform for each variant. This is calculated using the methods outlined below, and it will show how often a certain variant will likely convert better than the baseline.
Results with a sample size above 30 and chance to outperform of greater than 97.5% are assumed to be significant.
In our example, we can see that both segments has a really low chance to outperform over the baseline, since both segments negatively impacted our baseline.
% Chance to Outperform - Calculation Breakdown
Chance to outperform is the probability that our variant (A) is better than our baseline (B). We use a method that follows Bayesian probability - calculation and source can be found below:
The probability of our variant (A) outperforming our baseline (B) is based on the distribution of the difference B – A. If the individual distributions of B and A are assumed to be normally distributed, then the difference B – A is also a normal distribution (Gaussian) with a mean of and variance of .
To find the chance of A outperforming B, we need to determine the area under the curve that falls to the right of zero.
The area under the curve or cumulative distribution can be expressed in terms of the error function erf, which has the mean μ and the variance σ.
Erf can be calculated with a numerical approximation, and we incorporate the same approach to for calculating chance to outperform:
Once erf has been determined, the final equation to calculate the chance that B is better than A is:
Note:Theoretically, the above mentioned method is Bayesian. However, you may have noticed that the math for this Bayesian method comes out to be similar to a normal two-tailed test as. we have seen as well. Since the the math turns out being pretty much identical to hitting a significance threshold, it can be said that chance to outperform is basically the same as statistical significance.
(Source: O'Connell, Aaron. “The Math of Split Testing Part 2: Chance of Being Better”)
The AB Test View will also display whether or not statistical significance has been achieved in the top left corner of the chart. Amplitude uses a two-tailed p-value of 95% confidence interval to judge whether or not a result is significant and only looks at the best performing variant. A 97.5% chance to outperform is the threshold for outperform significance. A 2.5% chance is also a significant result, but it signals a significant chance to underperform.
To help reduce false positives given continuous monitoring, we set a minimum sample size before we declare significance. Right now, we ensure that there are 30 samples and 5 conversions.
Sample sizes of less than 30 are automatically considered to be not statistically significant.
When a test has reached statistical significance, you will see the green text appear:
If the test is not found to be statistically significant, then you will see the following red text:
The data table below the chart will give you a breakdown of the data. As with all data tables in Amplitude, you have the option to export the data as a CSV file. Here are the columns included:
- Count: The number of users or groups that entered the funnel.
- Converted: The number of users or groups that completed all the steps in the funnel with all conditions met.
- % Conversion: This is calculated by the number of converted users or groups divided by the number of users or groups that entered the funnel.
- % Improvement over Baseline: This is calculated by the equation (% conversion for that variant - % conversion for the baseline) / (% conversion for the baseline). The percentage in the data table will be green when the value is a positive number.
- % Chance to Outperform: This is the percent probability that your test variant will convert better than the baseline using a model founded on Bayesian principles. The percentage in the data table will be green when the value is above 51%.
Conversion by Event & User Property
You can set up your funnel to view how users with an event property value at a specific step have converted through the funnel. This helps you analyze what property value potentially has the greatest or smallest impact on users' conversion through the funnel.
In the example below, we have a three-step funnel: 'Search Song or Video', 'Play Song or Video', and 'Share Song or Video'. In the first image, we've segmented the funnel by Step 2's event property, 'Content_Type'.
The below image shows the conversion distribution of users, who performed the 'Play Song or Video' event with that 'Content_Type' value. Of the possible values, we see that both songs and videos had roughly 42% conversion. Lastly, if you choose to break down by a Step that is not the first, you will see a segment of users who did not reach that segmented step (the orange shaded segment for 'did not reach step' in this example).
Note: If users in your funnel can complete the steps multiple times, then the "broken down by" will take the first occurrence of each event and bucket the user for the value on that event.