Why traditional A/B testing is slow
Learn why the general data model is orders of magnitude slower than what is possible.
1. High sample variance
Traditional A/B testing is unaware of the status you visitors: it doesn't know whether the samples are new- or return visitors or whether they had already converted.
Due to this high sample variance the results act wildly on the initial phases of the experiment, even if the variants are identical. You must wait for things to settle until both variants have roughly the same amount of visitor engagement.
Significant sample variance is the biggest reason why traditional A/B testing is slow.
2. Sampling from lagging indicators
Lagging indicators are the key conversion metrics on the bottom of the funnel: signups, customer conversions, and viral shares. Sampling from these metrics is slow because they happen so rarely. The lower down you go to the funnel, the fewer samples you get, and the more you must wait for statistical significance.
Things would be an order of magnitude faster when sampling from each page view. Which version made a better first impression and caused fewer people to bounce? Which variant was more engaging?
However, traditional A/B testing is not capable of measuring these leading indicators, and you must always wait for enough key conversions to occur.
3. Baseline data is missing
With traditional A/B testing, you must always collect the conversion data for both variants A and B. The waiting time would be halved if you only need to collect samples for the new, alternate version.
Unfortunately, all the data collected in traditional A/B testing is no longer usable after the experiment is over.