Creating A/B Tests in Yotpo SMS & Email Flows

A/B testing is a powerful tool that can boost the performance of your Yotpo SMS & Email flows. By testing variations of text messages and emails on a select group of customers, you can identify what content yields the best results. This enables you to refine your messaging, increase engagement, and improve flow effectiveness.

In this article, you’ll learn how to set up A/B tests for text messages and emails and monitor the tests’ progress.

How it works

In an A/B test, you can compare up to 4 variations per text message or email. Each variation can feature different content, design, or offer different discounts. To determine the winner, you can choose from the following metrics:

• Total orders: The total number of orders during the test period

• Total revenue: The total amount in USD generated from orders attributed to the message during the test period

• Click rate: The percentage of recipients who click a link in the message out of the total number of delivered messages during the test period

• Conversion rate: The percentage of orders out of all delivered messages during the test period

• Email open rate: The percentage of unique opens out of all delivered messages during the test period

Tests can continue until a specific date or until reaching a predefined target.

Setting up an A/B test for text messages

To set up an A/B test:

1. After you add a text message to your flow, turn on A/B testing in the flow builder.

2. Select the number of variations. The audience will automatically be split evenly per variation.

3. Specify the test duration. There are two options:

   • To set an end date

   • To allow the test to run until a target is reached. If you choose this option, you will need to define the threshold target of the test whose value will depend on the winning metric selected in the next step.

4. Select the winning metric.

5. Craft the content of each variation in the SMS composer.

The A/B test will start automatically once the flow is activated. The variation with the highest result by the end date or that reaches the threshold first will be the winner and will be sent to the rest of your customers. To learn more, see Winner selection.

Setting up an A/B test for emails

To set up an A/B test:

1. After you add an email to your flow, turn on A/B testing in the flow builder. By default, each test starts with two variations. To learn how to add more, see Adding variations.

2. Set the test duration. There are two options:

   • To set an end date

   • To allow the test to run until a target is reached. If you choose this option, you will need to define the threshold target of the test whose value will depend on the winning metric selected in the next step.

3. Select the winning metric.

4. Craft the content of each variation in the email editor. You can A/B test subject lines, preview text, and the email template content.

Your A/B test will start automatically once the flow is activated. The variation that achieves the highest result by the end date or reaches the threshold first will be the winner and will be sent to the rest of your customers. To learn more, see Winner selection.

Adding variations

By default, there are two variations, A and B. You can add 2 more by cloning Variation A or Variation B. The audience is automatically split evenly per variation.

To add a variation:

1. Select a variation by clicking its tab.

2. Click the 3-dot menu next to Email Variation Settings and select Clone. This action will generate a new tab for the cloned variation.

3. To add another variation, simply repeat step 2.
   

   

Winner selection

The variation that achieves the highest result by the specified end date or meets the threshold first will be the winner and will be sent to your remaining customers.

Once a winner is selected, its name will be displayed on the step’s card in the flow builder and within the A/B test settings.

When the test ends, its settings are automatically deactivated, the winning variation is saved and your customers will continue to receive it as part of the flow.

If you want to run a new test for the same step, re-enable the A/B Testing button.

Understanding A/B testing analytics

To gain insights into your customers’ behavior, visit the flows analytics page. It gives you access to A/B test results in real time. You can quickly get to this page by clicking the Review A/B test analytics link found in the SMS composer and the email editor after saving the flow.

Once on the page, scroll down to the Detailed Data section, and select the AB Testing tab. The data is organized in a table.

Please note:

The results of the A/B tests cannot be filtered by period. You will always see the data aggregated since the tests were created.

In the table below, you can find what data each column contains.