Inside the Sitecore Test Lab

A top to bottom review of Sitecore testing options

Sitecore's testing functionality is a powerful, yet seldomly used feature of the XP platform. With Sitecore 8, testing options are available directly from the Experience Editor, during the content editing session. It's clear that Sitecore is encouraging editors to test early and often. 

Despite the ease of use, it seems that the majority of marketers make little use of Sitecore's testing features. When talking with peers about this topic it seems that there are two main reasons Sitecore testing isn't part of every marketer's daily routine. 

#1. They don't have a clear and consistent engagement value scale to test against and don't have consistent goals across their website
#2. They haven't learned how to create, deploy and complete a test within Sitecore

This post focuses on the latter. 

SITECORE TEST TYPES

There are three main test types, page level tests, component tests and personalization tests. 

Page level testing enables you to test one or more variations of an entire page against another. Most typically, we're testing a variation against the original. This is commonly known as a/b testing. 

Component testing enables you to test one or more components on a page against another. When we test multiple components against one another in a single test we're talking about multivariate testing. Component testing can be done against the content - like marketing copy, imagery, videos, etc. (think data source). Component testing can also be done against the component or rendering variations. As an example, we may be testing the effectiveness of two different carousel renderings that both point to the same data source. 

Personalization testing occurs whenever component personalization is used. Trailing value per visit and reach metrics are available for every personalization variation. 

TEST OBJECTIVE

Why are we testing in the first place? We're testing as a mechanism to optimize the user experience. The measure of that optimization is one of two things. 

Goals are probably the most common test objective we work with. We may test marketing copy, call to action design or page layout against a single goal. That goal may be a demo sign-up, newsletter opt-in or advancement from cart to checkout. In this scenario, there's typically a direct expected causality between the test and the goal; within a click or two. 

Trailing value per visit is the other primary test objective that we use with Sitecore. This test objective uses Sitecore's notion of engagement value and uses the visitor's overall engagement value as a measure of test success. This is very useful when we're not optimizing for a very specific desired action (e.g., demo sign-up), but for a series of actions (e.g., view video, download PDF, read a blog) that, in aggregate can be a good indicator of customer engagement. 

TEST VARIABLES

If you open up a test in Sitecore (Marketing Control Panel > Test Lab) you'll find an inventory of variables that are housed in the test definition item. 

Test objective: This is what's described above. It's where you choose what we're testing for. It also lets you choose whether you want to select the winner manually or have Sitecore select it automatically at the end of the test. 

Test expectations: Tests have a selector that enables the user to make a guess at the impact of the test. Will it improve engagement, stay the same or decrease engagement. Right now, this is informational only, that is used in the experience optimization workspace - a little bit of gamification. 

Test strategy: We can choose a test strategy for our tests. The random strategy can show different variable to the same visitor. The sticky strategy shows only one test variable per user. 

Traffic allocation: We can choose the percentage of overall page traffic that will be exposed to the test. If we're testing something significant, like an alternate registration form, we may not want to expose all our visitors to that test. In that case, we push the form variant out to only 10% of our visitors. 

Confidence level: The confidence level relates to a statistically valid test result. The higher the confidence level, the larger the test sample size is needed. (-- insert SC's method for determining statistically valid tests)

Duration: This lets you put a minimum and maximum boundary around the test length. 

THE IMPORTANCE OF ENGAGEMENT VALUE

Without an engagement value scale you're going to be limited to testing against goals. In order to turn Sitecore into a testing machine (and to build a 'test everything' culture) you're going to need an engagement value scale. Once you have that in place Sitecore's reporting becomes more valuable, and opportunities for optimization surface. 

What do you do with a page that has high engagement value but low traffic? Look for ways to amplify that page across your site. 

What do you do with a page that has a lot of traffic but low engagement value? Look for opportunities to personalize and test content & presentation variations.