Why A/B testing is so slow
Learn why the general data model is orders of magnitude slower than what is possible with Volument.
Sampling from lagging indicators
Lagging indicators are the key conversion metrics on the bottom of the funnel: signups, customer conversions, and viral shares. Sampling from these metrics is slow because they happen so rarely. The lower down you go to the funnel, the fewer samples you get, and the more you must wait for statistical significance.
Things would be an order of magnitude faster when sampling from each page view. Which version made a better first impression and caused fewer people to bounce? Which variant was more engaging?
However, traditional A/B testing is not capable of measuring these leading indicators, and you must always wait for enough key conversions to occur.
No data for the original version
Let's say you push a new version of your front page and want to see how the new version compares to the original one. You would expect to have the original data available, but this is not how traditional A/B testing works. You must always start from scratch and collect data for both A and B. This essentiallly doubles the necessary waiting time.
And when the test is over, the data is no longer usable on your upcoming experiments.
Not understanding the funnel
Traditional A/B testing is unaware of the status you visitors: it doesn't know whether the samples are new- or return visitors or whether they had already converted.
Due to this high sample variance the results act wildly on the initial phases of the experiment, even if the variants are identical. You must wait for things to settle until both variants have roughly the same amount of visitor engagement.
