This article will help you:
If you are experiencing unexpected issues with creating and rolling out an experiment or flag, be sure to read on for a potential solution:
- The device ID and user ID you use for experiments and flags must be the same as the device ID and user ID used in Amplitude Analytics.
- In order for a flag to show up in a deployment, it must be enabled in that deployment.
- Do not assign a variant a name value of “off.” This value is explicitly reserved for users in the “OFF/FALLBACK” bucket.
- While it is possible for the assignment event and the exposure event to be the same event, they are different concepts, and each plays a different role in your experiment: The assignment event is the event that immediately precedes a user's assignment to a variant, while the exposure event is the event the user fires to be actually exposed to the variant. Users can be assigned at the beginning of their session, but don’t fire the exposure event until later in the flow.
Make sure you have the correct events assigned to each.
- IP geo-resolution is currently (June 2021) not enabled for targeting locations.
- Amplitude Experiment updates the targeting on cohorts each hour. Targeting on properties explicitly sent to Experiment is immediate and real-time. Targeting on user properties stores in Amplitude can be delayed by up to one hour, due to the CDN.
- Amplitude Experiment generates data on the server side and stores assignments as user properties. For this reason, in some cases a user may not see the variant to which they were assigned. The cause of this is usually timeouts, network errors, or ad blockers.
- If you are running an experiment on a single platform only (for example, iOS), you might still get users showing up with a different platform value (i.e., Android). This can happen if your users are using your product on multiple platforms, with their most recent event sent from a platform other than the one you've specified for your experiment. When the assignment event is automatically sent, it does not contain non-Experiment properties (like, for example,
To resolve the issue of syncing old properties, connect the Experiment SDK with the Amplitude Analytics SDK to ensure that user context is available.
- It's possible for a user to see one variant of your experiment one day, but another variant the next. Because bucketing in Amplitude Experiment relies on user ID and device ID, this can happen if you are relying on device ID alone and the user's device ID changes. This can occur if the user is using incognito mode, or if they've cleared their cache and cookies since their last visit.
- In some cases, a user can trigger the assignment event without logging any other active events:
- The user's adblockers or analytics blockers are interfering with analytics application events. But since experiment requests are made from backend servers, those generally will not be impeded. This mostly affects Web users.
- A user opened the app, a request was made to Amplitude's backend servers (which in turn triggered the assignment event), and the user left the app before firing any other actions that trigger an analytics event.
- Amplitude's servers are receiving assignment requests directly from your backend, instead of from the end user. Check if you have instrumented experiments in offline campaigns— sending a group push notification, marketing emails, etc.