AB Testing with User Predictions

As with all A/B tests, you will create a control group and a variable group. The variable group will experience the changes associated with the user prediction-based tactic that you are testing, while the control group will receive a neutral treatment, lacking the new tactic, and will serve as a basis for comparison.

As a result, you will be able to determine to what degree your Prediction Manager tactic drove additional conversions, purchases, or any other chosen metric. You can also test and compare multiple Prediction Manager (PM)-based tactics to determine which are most effective for you. Based on your test results, you can further refine your tactics and eventually implement winning tactics across your entire userbase.

For example:

  • In Site Personalization Manager:
    • Variable Group: If the user’s item_7 prediction (likely number of items within a purchase, if one is made over the next week) is greater than two, and the user has at least one item in their cart, display a different cart template that includes a list of five recommended items.
    • Control Group: Show your existing cart template page, regardless of the user’s item_7.
  • In Email:
    • Variable Group: Do not send the campaign to users with an optout_7 (likelihood to opt out over the next 7 days) k-tile value of more than 970.
    • Control Group: Send the campaign to all members of the control group.

Step 1: Create User Segments

  1. Create the control group:
    1. In Audience Builder, select Generate List.
    2. For your Source List, choose Primary Lists (or any list that represents the full set of users that you want to include in your test).
    3. Enter a List Name, for example Test – Control.
    4. Enter a Random Selection Size for the list (for example, 50%).
    5. Click Submit Query.
  2. Create the variable group:
    1. In Audience Builder, select Create Smart List.
    2. For your Source List, make the same selection as you did when creating Primary Lists.
    3. Enter a List Name, for example Test – Variable.
    4. Under Criteria, select “Is not a member of list,” and in the Value field, enter the exact name of your control group list.
  3. Create a Smart List to capture future signups, who will not be part of the control or variable groups. It will be your choice whether to give this group the old (control) or new (variable) treatment. However, it is essential that you do not include these users’ results when you are comparing the control and variable groups, especially given the typically higher engagement of new users.
    1. In Audience Builder, select Create Smart List.
    2. For your Source List, make the same selection as you did when creating both prior lists.
    3. Under Criteria, select “is not a member of list” and in the Value field, enter the variable and control list names separated by a pipe, in order to include all users who are members of neither list. For example, Test – Control|Test – Variable.

Step 2: Vary the Treatment Per Segment

Option A: All Users in Variable Group Receive a Template that Includes PM Logic

PM AB test 1

  1. Using a use case above, or your own custom Prediction Manager tactic, create a pair of email or SPM templates:
    • A control template, without the Prediction Manager logic you are testing (which can be an existing template), and
    • A variable template, that contains the Prediction Manager logic you are testing.
  2. These templates should differ only in that the control template lacks the Prediction Manager tactic you are testing.
  3. If testing an Email-based tactic, generate three identical campaigns, one for each test segment list. The only difference among the campaigns: the variable segment (and optionally the new user group) will receive the test template.
  4. If testing a SPM-based tactic, create two audiences for the same section–“Control” and “Variable”–and set each to view the applicable control vs. variable template.

Option B: Prediction Manager Criterion is Used to Further Segment the Variable Group Only

PM AB Test 2

For example, let’s suppose you wanted to send a campaign to all users, and test a follow-up message with a coupon sent only to customers in the top 10% most likely to purchase (purchase_7 > 900) to determine how offering this ‘VIP’ coupon affects purchase rate, while ensuring that it does not increase optouts. Unlike test type A, you will not send a different template to the control vs. variable groups. You will instead create a sub-segment within the variable group based on a Prediction Manager criterion, which will determine which of those variable users will receive the coupon.

  1. In Audience Builder, select Create Smart List.
  2. For your Source List, select the variable list (for example, Test – Variable).
  3. Enter a List Name, for example “Test – Variable – High Purchase,” and include the “purchase_7 is at least 900” criteria.
  4. Send the same initial campaign to both control and variable lists.
  5. Send an additional follow-up campaign, containing a VIP coupon, only to the “Test – Variable – High Purchase” list.

Step 3: Compare Segment Metrics

  • Use Campaign Detail Reports to compare the performance of your campaigns to identify which tactic was more effective, according to your choice of metrics. For example, you may seek to answer the question: did the test tactic increase purchases without increasing optouts?Note: Be sure to measure for statistically significant results: use sufficiently large sample sizes and identify only significant differences between control and variable outcomes. If you have questions, please contact your customer success manager.