Why traditional A/B testing is so slow

Learn why the general data model is orders of magnitude slower than what is possible with Volument.

1. High sample variance

Traditional A/B testing is unaware of the status you visitors: it doesn't know whether the samples are new- or return visitors or whether they had already converted.

Due to this high sample variance the results act wildly on the initial phases of the experiment, even if the variants are identical. You must wait for things to settle until both variants have roughly the same amount of visitor engagement.

4,400+ visitors and 20 days was needed to show that the versions were identical 🤷

Significant sample variance is the biggest reason why traditional A/B testing is slow.

2. Sampling from lagging indicators

Lagging indicators are the key conversion metrics on the bottom of the funnel: signups, customer conversions, and viral shares. Sampling from these metrics is slow because they happen so rarely. The lower down you go to the funnel, the fewer samples you get, and the more you must wait for statistical significance.

Things would be an order of magnitude faster when sampling from each page view. Which version made a better first impression and caused fewer people to bounce? Which variant was more engaging?

However, traditional A/B testing is not capable of measuring these leading indicators, and you must always wait for enough key conversions to occur.

3. Baseline data is missing

Let's say you push a new version of your front page and want to see how the new version compares to the original one. You would expect to have the original data available, but this is not how traditional A/B testing works. You must always start from scratch and collect data for both A and B. This essentiallly doubles the necessary waiting time.

And when the test is over, the data is no longer usable on your upcoming experiments.

{"short_desc":"We mean slooooooooooow.","desc":"Learn why the general data model is |orders of magnitude slower| than what is possible with Volument.","title":"Why traditional A/B testing is so slow","url":"/blog/why-traditional-ab-testing-is-slow","key":"why-traditional-ab-testing-is-slow","created":"2019-11-15T05:12:04.007Z","modified":"2020-07-24T04:40:47.628Z","createdISO":"2019-11-15","modifiedISO":"2020-07-24"}