What is A/B Testing?

What is A/B testing, and how does it work?

A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page or product feature in order to optimize conversion rate, or improve upon a certain business metric. The two versions can be very similar, with only a change in button color, or very different, with a total change in the way a feature behaves.

A/B testing and A/B/n testing are based on the scientific method, and the process is very similar. To start with, gather relevant data on your current features and see which ones have the most potential to improve key business metrics. After you have the baseline data, look at those features to see how customers are utilizing them, and hypothesize a variation which could improve it.

Since the next step is to build the new version, you'll want to take improvements with similar expected user experience improvement and prioritize them by how easy they are to build. Then, pick a test to start with and build the new version. The old version will be what scientists call the "control" and what we'll call Version A; the new version is the "experiment": Version B.

With front end A/B testing, people typically assign Version A and Version B to different sets of users and measure which set of users, if either, had a higher conversion rate with statistical significance. But there is a different kind of A/B testing which happens at a much deeper level.

A/B Testing With Feature Flags

While typical A/B testing happens on the front end, choosing which version of the page show to website visitors, there is a way to A/B test any product feature, whether front end or back end, using feature flags.

Feature flags allow development teams to release a feature to only a subset of users, which satisfies the necessary step of creating two versions of a feature. All that's left to do is to integrate the team's analytics platform with the feature flag management system, such that the team can correlate the users' behavior with which version of the feature they used. Mature feature flag management systems have built-in analytics capture and pre-built integrations to analytics platforms, simplifying this integration.

After these things are done, the A/B testing process can be used to measure the impact of the user experience change when any new feature or code change is implemented. Development teams can then look at this information and adjust the feature accordingly: if the change is significantly negative, they can find out what's wrong and roll back the feature so it performs as it did before running the test; if it's positive, they can release the product feature to a larger percentage of their customers.