Multivariate Testing
05.10.2009

Companies can achieve dramatic improvements in conversion rates and sales by using multivariate testing to scrutinise customers’ online journeys
Quick Facts

• Multivariate testing (MVT) can improve conversion rates by as much as 30%
• 39% of companies are dissatisfied with their conversion rates
• 39% of surveyed companies intend to apply MVT
• Positive chemistry between the supplier and client team is a critical success factor

It’s every site owner’s nightmare: your brand new website has been user tested and overhauled to showcase popular products and create more effective customer journeys, but when the site goes live conversions start falling. What do you do?

In situations like this, there could be so many variables affecting performance that testing each of them would take too long. For example, standard A/B or split testing will determine the more effective of two combinations. But if you wanted to test the effectiveness of your customer order journey across a number of pages, A/B testing could take considerable time. This is where multivariate testing, or MVT, comes in.

As sites have become bigger in terms of pages and traffic, MVT has become standard practice for many web marketers. Of the companies surveyed last October by Econsultancy for a conversion report released in association with RedEye, 17% currently apply MVT and 39% plan to do so soon. The reasons are compelling. MVT enables you to test the effectiveness of multiple web page combinations at the same time. You can ascertain which variations and sales journeys best meet your site goals.

Suppose you set up three MVT paths. Visitors to your site would be directed along one of the three paths so that each got equal amounts of traffic. You may then decide to test several related pages per path and several areas per page; the heading, copy and imagery, for instance. Theoretically, MVT can cope with limitless combinations, restricted only by needing sufficient visitor volume to obtain statistically valid information.

It’s not only websites which benefit from MVT. Email marketers use the method to optimise campaigns by evaluating different images, banners, calls to action, promotion and prices. The discipline also extends to mobile platforms where static and dynamic components such as image size, selection, content, placement, design, graphical elements, headlines, colours and functional variations can all be tested.

In Econsultancy’s report, 39% of companies admitted conversions didn’t meet their goals. Dissatisfaction with site conversions is a big issue for many companies. For those transacting much of their business on the web, it translates into hundreds of millions of pounds of lost income. Thus testing has become a priority.

But there was also positive news. Of those companies which succeeded in improving their performance, market segmentation by demographic, geography and behaviour had an improving effect. Customer journey analysis also had a positive effect and testing was recognised as a major contribution to website performance.

Critical analysis

The results from MVT can be impressive. It’s not uncommon to see conversion improvements reaching 30% or even 40%. But given the number of MVT tools available, the real challenge for ecommerce practitioners is which to use. Tools include the Google Website Optimiser, Omniture, SiteSpect, Optimize and Maxymiser.

Sometimes there’s no choice but to implement MVT if conversions take a dive. Jenny McKenna, ecommerce manager of Fragrance Direct, was presented with just such a problem. Her company sells brand perfumes at discount prices. It enjoys annual growth of between 20% and 30%, which necessitates continual enhancement to its website hierarchy, features and functionality.
In September 2008, Fragrance Direct replaced its old site. The new site had greater functionality, added features and a new purchase form. It was designed to show products more attractively than before and to make the customer sales path easier to navigate. But the site delivered anything but its objectives.

Conversions dropped and growth stopped. Like most retailers, Christmas is Fragrance Direct’s busiest period. Yet 2008 Christmas sales were significantly lower than the previous year’s. User testing had worked well and there had been no indication of possible issues. What was known was that minor or subtle changes can have a major impact on conversion rates.

For this reason, McKenna adopted MVT. She considered Google Website Optimiser, which she rated highly, but decided that she also needed support, consultancy and a high level of analysis. She reviewed a number of MVT tools and ultimately appointed Maxymiser in February 2009. She needed it to generate some quick wins.

Maxymiser proposed immediate and seemingly trivial changes to fonts, colours and product page copy positioning. It suggested changes to the presentation of messages such as ‘in stock’ or ‘out of stock’, ‘savings’ and ‘special offers’. It also made changes to the order path.

McKenna got her first quick win. The alterations generated a 33% improvement in completed journeys from a product page to check-out. In March, testing and analysis started in earnest. In April, conversion rates improved significantly and again in May, and the year continued with incremental monthly improvements.

Fragrance Direct now enjoys a substantial uplift in conversions and revenues. McKenna believes that a key element in MVT’s success was the relationship between her team and Maxymiser.

Result variance

Of course, MVT requires a good volume of users to produce the statistics required to signpost optimum sales routes. Double-glazed window, door and conservatories retailer Anglian has a website which lends itself to MVT, as it relies on it to generate quality sales leads.

With a view to improving visitor-to-lead conversions, Judi-mae Galer, Anglian’s ecommerce co-ordinator, worked with digital agency Latitude to apply A/B testing and MVT using Google’s Website Optimiser. In January 2009, she applied A/B testing to Anglian’s conservatory products. This was completed in February and resulted in an increase of only half a percent, so Anglian decided the conservatory area of the site needed to be redesigned.

However, in March A/B testing for Anglian Window products generated a 98% increase. In April, applying Google Website Optimiser enabled Anglian to generate a further increase of 27%. And in September, after a redesign of the conservatory pages, A/B testing resulted in an uplift of 108%. MVT testing is about to be applied to Anglian conservatory product pages.
“Testing is a vital part of successful ecommerce,” says Galer. “It should be a continual process. And given our MVT results, I would recommend Google Website Optimiser.”
Finance is another market which can derive significant benefit from testing, analysis and website enhancement. Matt Douglas, senior on-site optimisation manager at Alliance & Leicester, is responsible for both A/B and multivariate testing of A&L;’s website. The Santander-owned bank uses its own A/B testing method to try to increase its market share, but the process generated conversion improvements at a slower rate than desired so 12 months ago it decided to introduce MVT.

Douglas opted for Maxymiser, which offered a combination of technology and consultancy. He also remarked on a recent Maxymiser innovation, One Touch, which minimises Alliance & Leicester’s IT support needs. It had already piloted tests with Maxymiser a year before and its selection criteria involved factors including functionality and cost of implementation. During this process, Douglas looked at Google Web Optimiser, which he rated as a great tool but with analytical limitations.

Where applied, MVT has given Alliance & Leicester a conversion rate uplift of between 10% and 15%. This translates into huge sums. “Don’t promote your results in the early part of the test,” advises Douglas. “They often look great to start with but the uplift needs to be sustained. Use full end-to-end tracking. Use your back-office systems to track what happens after the submit button has been pressed. This extra level of information is extremely valuable. Check with your colleagues that nothing will change on the site during the test period.”

Given the significant benefits of multivariate testing, it’s clear it will remain at the top of the business agenda for the foreseeable future.

Case study: BT.com increases conversions by 37%

Jennie Wright, BT’s senior marketing manager for the Consumer Online division, is a keen advocate of multivariate testing. She has both a technical and commercial background and has detailed awareness of what MVT can deliver. Wright evaluated five MVT suppliers and shortlisted three, giving each a BT.com control page to test.

She had four evaluation criteria. First was effectiveness: whether the MVT technology and the supplier consultancy could deliver on their promise. Second was technology: would the supplier’s technology match BT’s from an implementation perspective? Third, analysis: could the supplier offer in-depth knowledge of MVT over and above BT’s. Finally, was the relationship: how good was the chemistry between the supplier and the client team?

Wright ran three proof-of-concept trials and briefed BT.com’s retained digital creative agency Crayon to collaborate on a test plan with two of the suppliers and build assets. Crayon and the suppliers provided post-test analysis.

Wright’s focus was on BT Vision, broadband and packages. Her objectives were to improve conversion and achieve sustainable improvement. Crayon’s approach was to organise the test plans so BT could test five areas per page and four variations of each area.

Crayon analysed page combinations, order journeys and sales paths. It looked at features, benefits, variations, images and messages, then analysed new and returning visitors and those coming from affiliate websites. Visitors behaved differently depending on which category they were from. A hundred combinations were tested over the first four weeks.

The result was that the best performing route generated a 37% higher conversion-to-sale rate than the control page. Chris Michael, commercial partner at Crayon, says, “The MVT process can improve the customer journey. All you need do is switch off routes that don’t get you the results you seek during the testing. You don’t have to wait until the end of the process.”