How to eliminate randomness and make better decisions In web analytics
I will admit statistics is hard.
I’ve always considered myself a creative marketer and a storyteller, and as soon as I start reading up on statistics and data science, my mind starts drifting.
But in marketing, you have to deal with numbers, and you need website and marketing data to optimize your efforts. To succeed, you don’t have to be a statistical ninja, but there is one concept you should understand.
When I worked as a marketing consultant, some clients would ask me things like:
“So, we got 20 visits from Facebook. Is that good or bad?”
“Yesterday, our conversions increased 800% from 1 to 9. What does that mean?”
“We updated our website yesterday, and our organic traffic is down 30% today. What is going on?”
“12 people have downloaded our ebook. Are we onto something here?”
No. No. No, and no.
The one concept you should wrap your head around is statistical significance.
I’ve talked with many marketers who have no idea what this concept means.
The gist is if you use data for decision-making, you have to understand whether the data you’re working with has value or significance.
What Is Statistical Significance?
Disclaimer, this is a deep and complicated topic. The goal of this article is to uncomplicate the idea for anyone who wants to understand how to use website data to make better decisions.
Let’s say you create new content for your website and want to understand whether the content is good or bad. Or, you create a Facebook campaign and want to know how that campaign performs. Or, you decide to do some A/B testing for a product page and want to see which variation resonates better with site visitors.
To answer these questions confidently, you need sufficient data or a big enough sample size.
If twelve people download your ebook and later signup for your service within any time range, you can’t judge whether that ebook is good or beneficial for your business.
Determining something good or bad based on a very small dataset is an opinion, nothing more.
Statistical significance tells you whether the findings and results are relevant or pure chance and whether one factor - a new piece of content - affects another factor - conversions.
The maths is pretty simple: a bigger sample size is usually better than a small one. But you don’t need all the data you can get because that takes time and money. You need a good amount of samples.
When working with bigger sample sizes, “you’re less likely to get results that reflect randomness.”
What is a good amount?
In Volument, we want to keep things simple and make analytics more open and accessible to everyone.
So when we deliver insights and tell you whether something works or doesn’t, we eliminate the randomness and ensure you have a statistically significant dataset.
For us, a good amount or large enough sample size is 100, and most statisticians agree this is a good number for a minimum sample size.
So, for example, before we give any insights into how a particular piece of content converts people to signup for your service, we require 100 conversions.
This helps us deliver insights that are relevant. You can learn more about this in How it works.
The pitfalls of biased data
If somebody tells you they are not biased. They are lying.
As humans, we make tons of biased decisions daily. Just think about all the data you filter daily and make everyday decisions based on bits and fragments of information.
In business, we have to be a bit more careful.
Data bias is another concept I want to mention here because if you don’t have a large enough sample size, you will likely overweigh elements in your data.
Any bit of information can be used as an agenda amplifier; you know the story you want to tell and find the data supporting that story.
Agenda amplification is a big problem in web analytics, and some popular metrics support this thinking. UX guru Jared Spool makes a compelling argument about how, for example, bounce rate can act as an agenda amplifier:
“Bounce rate. Bounce rate is the most-cited statistic by people who are trying to validate their content decisions. ‘Our bounce rate is high, so we need to write better content.’ Or, ‘Our bounce rate is high, which means people are coming and finding out exactly what they want. Our content’s good enough.’ You pick which side of that argument you’re on, and then you can interpret bounce rate to support any argument you want.”
Data storytelling is crucial in marketing and web analytics. But, you should be critical of the material you use to build the plot because that will decide whether your story becomes fact or fiction.
Join our waitlist if you want to eliminate randomness in web analytics.