Running an A/B Test on Your Email Campaigns

Test different content in your email campaigns so you can better understand what works best for your audience. By running A/B tests on your email campaigns, you can determine what content will work best at increasing your open rates, click-through rates, and/or conversion rates.

In this article, you’ll learn how to set up and run A/B tests on your email campaigns.

How it works

The flow for creating and running an A/B test is as follows:

1. Create an email campaign with up to 3 different variations. 
2. Define what percentage of your selected audience you want to participate in the test. Any audience members who do not participate will receive the winning variation at the end of the test.
3. Choose a winning metric - this is the metric that decides which variation wins.
4. Define how long the test should run for. You only need to do this if you do not want all audience members to participate in the test.
5. View aggregated analytics on the performance of your variations, including click-through rate, open rate, bounce rate, and more.

Step 1: Create your A/B test

To create your A/B test:

1. In your Yotpo SMS & Email main menu, go to Campaigns > My Campaigns.
2. Select the campaign you want to run an A/B test on, or create a new one by clicking Create campaign > Email.
3. Set up your campaign. Learn more in Creating an Email Marketing Campaign.
4. In the email editor, scroll down to the Email Content section.
   
5. Click Create A/B test. A pop-up appears.
   
6.  Add or remove variations by clicking the icons next to your variations:
   - To add a variation, click the Duplicate icon
   - To rename a variation, click the Edit (pencil) icon
   - To remove a variation, click the Delete icon
   
   Please note:

   You can create up to 3 different variations of your email content. You’ll be able to edit the content of your variations later.
7. By default, your variations will be divided equally among your recipients. If you prefer to test on a smaller percentage of your audience, drag the Winner slider to the right. This will decrease the number of recipients who participate in your test and instead send the winning variation to the remaining percentage of your audience at the end of the test. 

   Example: If you set the Winner to 60%, then 20% of your audience will receive Variation A, 20% will receive Variation B, and 60% will receive the winning variation at the end of the test.

8. (Only if you moved the Winner slider): Define the test duration. This is the number of days/hours that the campaign will run before the winning variation is sent to your remaining recipients.
9. Select your winning metric from the following options: click-through rate, conversion rate, open rate, total orders, total orders. The winning variation will be decided according to its performance in this metric.
10. Click Create test. Your A/B test won’t start until you launch your campaign.

Step 2: Edit your variations

Navigate between your different variations by clicking the dropdown arrow next to the variation you’re currently working on.

Remember to save your changes.

Step 3: Configure your test settings

You can change your winning metric at any time before you send the campaign. Just click the dropdown next to Email Content and select Manage A/B test. After the test starts, you won’t be able to make any changes to your test settings.

Step 4: Analyze your test results

When your test has finished running, you can analyze its performance and see which variation was the winner.

1. In your Yotpo SMS & Email main menu, go to Analytics > Campaign Analytics.
2. From the list, identify the relevant campaign, and click View.

From your Campaign Analytics, you can see the following:

• The performance of each variation according to the winning metric
•  An indication of which variation is the winner
  
• The campaign performance of each variation, along with its deliverability analytics

Best practices

Test one variable at a time

For example, if you edited the subject line, don’t change the preview text. This will make it easier to identify which variable contributed to improved performance.

Wait until you have a good sample size of viewers before you draw conclusions

We recommend you let your campaign run for a few days before you start analyzing the results. This will allow you to gather a sufficient amount of data to make an informed decision about which version of your email campaign is performing better.

FAQ

Is there a limit on the number of A/B tests I can run?

No, you can run as many A/B tests as you like!

What can I test?

You can test your email subject, preview text, and/or email content.

How many variations can I test?

You can test up to 3 variations at a time.