Running an A/B Test on Your SMS Campaigns

Our new and improved A/B testing experience for Shopify merchants is here! Users will now be able to“pre-test” their campaigns and send their subscribers the top-performing message, which is most likely to convert.

With A/B Testing, Shopify store owners choose a test group size, number of groups to test against each other with different text messages. The winning message will be sent to the remaining subscribers outside of the test groups. The winning value, which determines the winning message, can either be the number of link clicks, orders made or revenue generated.

Setting up A/B testing

1. To set up your A/B testing environment, you first need to enable it from the toggle button.  

2. Number of test groups is the number of separate groups of subscribers that will be created to compete against each other.

3. Your Test group size is the percentage of your chosen target audience that will be included in the test. The larger the test group size, the smaller the winning group.
   Advice: Use a lower test group size, i.e. 20% to maximize conversion rates. The winning message will send out to the remaining 80% of subscribers outside of your test groups.

4. Test Period is the duration of the A/B test. Enter a value in minutes, hours or days to set how long the testing will run before a winning message is determined.

5. Winning metric is the determiner that decides the winning test group. It can either be set to number of link clicks, order, or revenue. The message with the greatest value generated will trigger a win-case scenario to send the winning message. ​

Writing your messages

You should write a different text and offer a unique promotion for each message according to the number of test sets you set. Be aware that an effective A/B test message will test only one variable, so focus your attention solely on the copy of the text, the type of discount offered, or an engaging image/GIF. This approach will let you isolate what variable worked best.

When writing your message, keep in mind the following:

• Adding an image or GIF allows for a 1600 character length, while emojis reduce it to 66.

• Only use the short-codes that are available. Pasting short codes from other campaigns will not work.

• Click the add STOP to opt out checkbox to be compliant and give subscribers the ability to unsubscribe from your SMS marketing service.

Review and send

After you are done writing your message, click on the Review Campaign button to look over the details before sending.

Make sure that you see all green ticks under Campaign Overview. This means that you have filled in all the requirements correctly, and your A/B testing campaign is ready to launch.

If you would like, click on Edit to change a setting.

Finally, you should read over the details and cost in Campaign data.

Click on the Send Now button to launch your A/B Testing text marketing campaign.

Once the winning message is determined, it will send to the remaining percentage of SMS subscribers outside of the test groups.

After the campaign is sent, you can track how well it did in the Analytics section of the app. After a bit of A/B testing, you will soon realize that the value of sending an effective A/B testing campaign, is maximizing customer engagement through the best-crafted message.