A/B Split Testing Transactional Emails
Continuously improve the design and creative of your transactional emails by A/B testing different versions. What subject line is most inviting? What’s the best place to place a link to get an invited user to join your site?
The basic idea of A/B testing is simple. Create two slightly different versions of a transactional template and set up to send using the API. Reporting will determine which one gets a better response from your users, usually measured by click and open rates.
For more information on mass mail campaigns, see A/B Split Testing Campaigns.
Setting Up the A/B Test
From inside the Template Editor, click the A/B Test button and hit OK at the confirmation prompt.
The names of each template will be appended with A and B. You’ll be inside the template editor of version A. To view both, save and click out to the Templates tab in the main menu.
Using the Rename button to change the name of a template to end with [A] or [B] does not make it an a/b cell; the A/B Test button must be used.
You can give it a unique subject line as well as change the message content.
A/B Testing in Action
For a given transactional email sent via the API, Sailthru will pick randomly between the different A/B versions. After a period of time, you will be able to compare the results and determine which is the most effective template.
Note: A user’s placement in the A or B grouping will persist for all future transactional test sends.
Reporting on Testing
In the Reports tab, click on Transactional A/B Testing Report to see analytics on the different versions.
Once you’re satisfied that you know which is the better performing template and you want to stop testing, reach out to your Customer Success Manager and they will mark the template you no longer want to send as inactive. Whichever version is left will be sent from this point forward.
Note: If you mark both versions as inactive, Sailthru will send neither. Do not delete the template as you will lose its historical performance.