Creating A/B Tests for Subscriber Collection Popups

Running an A/B test on your subscriber collection popups lets you compare their performance to find the most effective ones. This helps you improve customer experience and boost conversions.

In this article, you’ll learn how to set up A/B tests for popups.

How it works

Yotpo SMS & Email allows you to compare up to 4 popups or spin-to-win forms in one test. After adding the variations, you will be able to adjust the audience distribution for each and select a metric to determine the winner. You can choose to run the test until a specific date and time or end it manually.

To streamline the process, you can conveniently customize all variations within the popup editor.

With popups being automatically integrated into flows to call customers to action, popup A/B tests have an impact on the flows you’re using. After you start a test, new flows will be automatically created in Flows > My Flows. To learn more, see How popup A/B tests affect flows.

Please note:

You can only compare forms of the same type: popups with popups, and spin-to-win forms with other spin-to-win forms.

Setting up an A/B test

To set up an A/B test:

1. In your Yotpo SMS & Email main menu, go to Audience > Subscriber Collection Tools.

2. Find the popup you want to test in the list, click the 3-dot menu, and select Create A/B test.

Step 1: Choose a test name

Every test comes with an auto-assigned name but you can change it in the Test name field. The test’s name won't be visible to customers at any point.

Step 2: Select the popups for comparison

Use the dropdown menus to select the popups to test. They can be with status Draft or Live. To add a variation, click Add popup.

If you want to test different variations of the same popup, simply select the popup and a duplicate variation will be automatically created. You will be able to customize the variations before starting the test.

Please note:

After a test begins, all popups, including drafts, will go live for the duration of the test. If you add live popups to a test and save it as a draft, the popups within it will also become drafts. Be careful not to accidentally remove all live popups from your store.

Step 3: Distribute your audience

Define how many people from your audience should see each variation. By default, each variation will be shown to an equal number of people but you can uncheck the checkbox and use the slider to split the audience as you wish.

Step 4: Select the winning metric

Select the metric that will determine which variation performs best: SMS CVR, Email CVR, Average Form CVR, or Average Order Value. Conversion rates (CVR) indicate the percentage of the audience that viewed each variation and subsequently decided to subscribe using the popup. Average Order Value (AOV) is an attribution-dependent metric that reflects the average value of orders placed by customers who subscribed through each variation. Here’s a breakdown of each available metric:

Please note:

AOV requires accurate tracking of orders. Therefore, it is necessary to have a separate flow for each variation in the test. If you select AOV as a winning metric, flows will be created automatically for every variation to guarantee that orders are attributed correctly.

Step 5: Set test duration

Finally, define whether you’d like your test to run for a specific period or you’d like to end it manually. If you choose to specify a period, you can select what will happen to the winning variation. It can be automatically published on your website once the test is over, or you can have it saved as a draft.

Please note:

The test duration is scheduled according to your store’s time zone, which you can adjust in the app’s General Settings.

Step 6: Customize variations (optional)

If you want to edit the variations in the A/B test before launching it, click Customize variations. You will be taken to the popup editor, where you can easily switch between the variations using the dropdown menu in the top left corner.

Step 7: Start test

You can start the test from the popup editor or from the setup. In both cases, you’ll see a review of the test’s settings and be able to leave a note, a convenient option if you are managing multiple accounts. Notes are visible on each test’s analytics page.

Click Start test. Your A/B test will be listed on the Subscriber Collection Tools page. You can expand it to see details about each variation.

Managing A/B tests

After creating a test, you can manage it from the Subscriber Collection Tools page.

Editing variations

Editing a test’s variations is possible at any time. However, if you edit a popup that is part of a test in progress, it may affect the results.

Selecting a winner

You can end a test that is in progress at any time by manually selecting a winner. Simply click the 3-dot menu and choose Select winner. Choose the winning variation and decide whether to publish it or save it as a draft. Then, click End test. The remaining variations will be saved as drafts.

Important:

If there are variations with the same results at the end of a test, the winner is determined by the number of impressions per variation.

Viewing analytics

You can see detailed analytics for every A/B test that is in progress or has ended. Click A/B analytics and you’ll be taken to a dedicated page showing data in real time. To learn more about popup A/B testing analytics, see Understanding Popup A/B Testing Analytics.

Please note:

If you delete a test that is completed, you’ll no longer have access to its analytics page.

How popup A/B tests affect flows

By default, each new popup triggers the creation of a welcome flow, which is why A/B tests also affect how your flows operate. If a popup is included in a test, new welcome flows associated with this popup are automatically created in My Flows. The structure and number of flows depend on the metric you chose to determine the test winner. For more information, see Winning metrics.

These newly created flows allow you to conveniently manage the content of your messages for every popup variation. After the A/B test begins, the original welcome flows are deactivated, while the new ones become active.

Welcome flows for all sources

A welcome flow can be configured to send messages to new subscribers from any source: popups, keywords, checkout, etc. To achieve this, the step checking whether a customer subscribes through a popup needs to be deleted.

If a popup included in such a flow is added to an A/B test, the original flow will be deactivated, and new subscribers won't receive welcome messages. Instead, a new welcome flow will be created checking for subscribers who use that specific popup to subscribe.

Tip:

Before launching an A/B test, review your existing live welcome flows to make sure they will continue working as desired.

Winning metrics

CVR

If you select a CVR metric (SMS, Email, or Average Form), each variation is a separate branch in the flow.

AOV

If you select the AOV metric, a separate flow is automatically created for each variation to guarantee that orders are attributed correctly.

Important:

You can safely edit a flow’s actions at any time but it is highly recommended to avoid changing or removing conditions to ensure proper order attribution.

You won’t be able to deactivate or delete the automatically created flows once an A/B test goes live. After the test ends, the flows will remain active unless you delete them or change their status to Draft.