Why traditional A/B testing is slow

Learn why the general data model is orders of magnitude slower than what is possible.

1. High sample variance

Traditional A/B testing is unaware of the status you visitors: it doesn't know whether the samples are new- or return visitors or whether they had already converted.

Due to this high sample variance the results act wildly on the initial phases of the experiment, even if the variants are identical. You must wait for things to settle until both variants have roughly the same amount of visitor engagement.

4,400+ visitors and 20 days was needed to show that the versions were identical 🤷

Significant sample variance is the biggest reason why traditional A/B testing is slow.

2. Sampling from lagging indicators

Lagging indicators are the key conversion metrics on the bottom of the funnel: signups, customer conversions, and viral shares. Sampling from these metrics is slow because they happen so rarely. The lower down you go to the funnel, the fewer samples you get, and the more you must wait for statistical significance.

Leading and lagging indicators

Things would be an order of magnitude faster when sampling from each page view. Which version made a better first impression and caused fewer people to bounce? Which variant was more engaging?

However, traditional A/B testing is not capable of measuring these leading indicators, and you must always wait for enough key conversions to occur.

3. Baseline data is missing

With traditional A/B testing, you must always collect the conversion data for both variants A and B. The waiting time would be halved if you only need to collect samples for the new, alternate version.

Unfortunately, all the data collected in traditional A/B testing is no longer usable after the experiment is over.

Learn Volument