What to Know About A/B Testing vs Multivariate Testing

A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page, feature, user flow, or other resource in order to optimize for a metric or set of metrics (often conversion rate). Multivariate tests or MVT test, are run with a higher number of variables and generally provide deeper insight on how to optimize your page. In multivariate testing, your feature implementation becomes a combination of elements or customer experiences which can be decomposed and tested simultaneously.

Advantages of A/B Testing

Most examples of A/B testing involve the front end — testing different landing pages for purposes of conversion rate optimization — but all that’s really required for A/B testing is testing different ideas for a feature. It can just as easily be testing different features in software or mobile applications as testing different variables, including different headlines, different page elements, different versions of the page, design elements or redesigns, for user experience optimization.

Because there are only two versions of your software when conducting an A/B test, your tests will deliver dependable data quickly. The more variations you have, the longer it will take to gather significant data — the traffic required for A/B testing is much less than the traffic required for multivariate testing.

The first thing that needs to happen when you run an A/B test is to gather data on your current resources (homepage, feature, user flow, etc.) and figure out which ones aren’t performing as well as they should. This should be a list of the features that you would like to optimize.Once you have this list, you can prioritize them in order of the potential value that they could bring. You should work with both your data analyst and product person to gather this information.

Hypothesize different possible combinations of changes that will fix the problem. This is usually done by the product person. Segment your user base into two parts, showing the original page or feature to one part and the new version to the other. How quickly you get significant results depends on the amount of traffic you have, since for the test results to have statistical significance, they must come from a large population. After you have your results, if there’s a significant positive change, you can implement the new version for all users.

Testing Multiple Features

A/B testing can occur on more than two different variations (this is called A/B/n testing), but not much more than two. The mechanic of A/B testing doesn’t lend itself well to testing more than around four variations at once: testing a higher number of variations is slow, and even A/B/n tests won’t give you a granular view of which subtle changes actually produced the results.

What is Multivariate Testing?

Multivariate testing is a testing method using a statistical methodology using multiple variants that is similar to A/B testing in roughly the same way as a cube is similar to a square: same shape, higher dimensions. You can think of a multivariate test as multiple nested A/B tests. Industries such as ecommerce use multivariate testing extensively to test different elements of their website such as page designs, or different strategies for marketing campaigns for example.

Say you have two features that you’d like to test variations on instead of just one. If Feature A has two possible states (on and off) and Feature B also has two possible states, in a full factorial scheme, you end up with 4 different combinations: (A on, B off), (A off, B on), (A on, B on), and (A off, B off). Segmenting your user base into four parts this way will let you know which individual elements produced the change, and further, how much each one contributed.

How Do You Know Which Type of Test to Run?

The biggest problem with multivariate testing is that you need a high website traffic platform to have enough users coming through each group to achieve enough sample size and get significant results in a reasonable amount of time. Because the number of groups grows exponentially with the number of tested variations, you should be careful to make sure you have enough traffic to make a multivariate test worthwhile. So for a lower-traffic platform, it’s best to run an A/B test.

Learn More About A/B Testing

Interested in digging deeper on A/B Testing and Experimentation? We’ve got you covered:

• Learn about more about A/B testing and A/B/n testing
• Dig deeper into Multivariate testing
• And finally, check out the state of feature delivery in 2020

As always, if you’re looking for more great content like this, we’d love to have you follow us on Twitter @splitsoftware, and subscribe to our YouTube channel.

Switch It On With Split

The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Schedule a demo to learn more.

Get Split Certified

Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.