Running an A/B Test on Your SMS Campaigns

Goals and Outcomes

With this feature, you can optimize your Shopify SMS campaigns by finding the most effective message through A/B testing. By identifying and sending the top-performing message to the majority of your subscribers, you’ll have the opportunity to boost your conversion rates and achieve better results for your campaigns.

A/B Testing Overview

A/B Testing empowers you to pre-test your campaigns and deliver the most effective message to your subscribers. As a Shopify store owner, you decide the size of the test group and the number of groups to compare using different text messages. Once the test identifies the top-performing message—based on link clicks, orders, or revenue—it’s sent to the rest of your subscribers. This ensures you maximize conversions and drive impactful results with every campaign.

Setting up A/B testing

1. To set up your A/B testing environment, you first need to enable it from the toggle button.  

2. Number of Variations is the number of separate groups of subscribers that will be created to compete against each other.

3. Your Variation size is the percentage of your chosen target audience that will be included in the test. The larger the test group size, the smaller the winning group.
   Advice: Use a lower test group size, i.e. 20% to maximize conversion rates. The winning message will send out to the remaining 80% of subscribers outside of your test groups.

4. Test Period is the duration of the A/B test. Enter a value in minutes, hours or days to set how long the testing will run before a winning message is determined.

5. Winning metric is the determiner that decides the winning test group. It can either be set to number of link clicks, order, or revenue. The message with the greatest value generated will trigger a win-case scenario to send the winning message.

Writing your messages

You should write a different text and offer a unique promotion for each message according to the number of test sets you set. Be aware that an effective A/B test message will test only one variable, so focus your attention solely on the copy of the text, the type of discount offered, or an engaging image/GIF. This approach will let you isolate what variable worked best.

When writing your message, keep in mind the following:

• Adding an image or GIF allows for a 1600 character length, while emojis reduce it to 66.

• Only use the short-codes that are available. Pasting short codes from other campaigns will not work.

• Click the add STOP to opt out checkbox to be compliant and give subscribers the ability to unsubscribe from your SMS marketing service.

Review and Sending

After crafting your message, click Review Campaign to double-check the details before sending it out. Look for all green checkmarks under Campaign Overview, which confirm that you’ve met all the requirements and your A/B testing campaign is ready to go. If you want to adjust anything, click Edit to make changes.

Next, review the details and cost in the second part of the Campaign Overview section. When everything looks good, click Schedule campaign to launch your A/B testing text marketing campaign. Once the test identifies the winning message, it will automatically send to the rest of your SMS subscribers outside the test groups.

After your campaign is sent, head over to the Analytics section in the app to monitor its performance. With a bit of A/B testing, you’ll quickly see the value of optimizing your campaigns—delivering the most engaging message to maximize customer impact!

Additional Information

SMS Deliverability Issues and Troubleshooting Steps

SMS deliverability issues can happen for a variety of reasons, such as delays from the provider, carrier problems, network connectivity, roaming, handset issues, or invalid numbers. However, just because a message failed once doesn’t mean it won’t succeed the next time. Try sending the message again, or test with a different number to see better results.