The future of website analytics and A/B testing

With the introduction to Volument we’ve seen a spike of interest towards privacy-friendly analytics focusing on conversion optimization. After working on the problem for several years we’ve seen many unexplored ways to benefit from data. Here’s our longer-term plan to truly change the analytics game.

Let’s start with the core use-cases of analytics:

1. General analysis: browsing data and taking insights
2. Marketing campaigns: market discovery and user acquisition
3. Website optimization with A/B testing
4. Product analysis

Product analysis is out of our scope and we are happy to hand that problem to other great products like Heap, Mixpanel, or Amplitude. The first three tasks, however, are the key use-cases of Volument. This article shows you how these functions can be drastically improved.

General analytics turns automatic

We’ve seen that a typical way to use analytics is fire and forget: Just set up tracking so that the data is available when needed. People tend to have other things on our mind than website analytics. With all honesty, we behave the same way. Regardless of the data, features, or how beautifully the information is designed.

However, things get a bit more interesting when vital information is automatically pushed in front of your eyes. Especially when it’s something urgent that directly impacts conversions.

We’re building an intelligence layer on top of the core traffic and traction data to detect anomalies and to send notifications to Slack, email, and to any configured server.

With automation in place, the analytics software does all the analysis work for you. When Volument is silent: you can be sure that things are running smoothly. That is: no news is good news. This is definitely not a new idea, but you don’t see great analytics solutions out there that take full advantage of automated self-analysis.

The above alert is just a small example of what’s possible. The true power comes when the intelligence layer is coupled with marketing campaigns and A/B testing. Let’s look closer.

Automated marketing campaigns

What is a marketing campaign? Is it a report for some landing page? Or is it analysis for a specific utm_campaing tag? Yes: those two are the most common approaches.

However, a campaign can be much more:

This is a fully automated system that detects spikes in traffic and compiles all the details on what happens while the traffic is peaking.

Let’s look at the benefits:

1. It’s all automatic: you’ll get a notification when the spike is detected when the initial performance results or “leading indicators” are available, and when the campaign is ended and the final traction numbers are available.
2. Know everything about the campaign: the referral domains, landing pages, the continents, and the countries. Most importantly, you get all the traction data: the details on awareness, interest, desire, and actions — as well as the profits, and losses.
3. Clear campaign history: see all your traffic spikes annotated and documented without extra effort. There’s no endless amount of UTM rows to hide the signal from the noise.
4. Compare all the campaigns against each other in an apples-to-apples fashion (using Volument’s unique data model) regardless of when they occurred. Eventually you learn what works and what turns people away.
5. No lost data when the UTM tags are stripped out from the URL when shared in social media. You also get more viral spread when your links are free from UTM clutter and other tracking parameters. (UTM tags are an absolute no-go in Hacker News for example).
6. Non-campaigns. Sometimes your traffic can spike like magic, without you doing a thing. You are ranked better in Google search, someone major points to your site, or an affiliate link is created. Volument auto-detects the reason for your spikes.

Automated campaigns are implemented later this year.

Automated A/B testing

A/B testing is another area where automation helps tremendously. Before jumping into the solution, let’s summarize the problems in traditional A/B testing:

 

• It takes ages to get statistically significant results. Even with decent traffic (100K monthly sessions) you literally need months to wait before you know which button is better.
• You are not supposed to make any changes on the website while the test is running or the results are distorted. This is, of course, out of the question for any serious business that wants to develop and grow.
• You can only measure the impact on one metric, typically at the very end of the funnel. Everything else is ignored: how the change impacted first impression, and the desire to move onwards the funnel. Even if you have the patience to wait for the final result, it may not mean a thing.
• You can only test minor changes, like button colors and alternate headlines. You’ll focus on irrelevant stuff rather than fixing real problems. Just ask yourself: how much a button color really matters? Yes — it doesn’t. If you don’t believe it, please wire up a test and wait half a year for the results without making other edits on your site. LOL 🙂

It’s not a surprise traditional A/B testing has failed to deliver the promise.

The solution

Here’s a significantly more powerful system for A/B testing to be introduced in Volument:

 

 

This is a fully automated system that monitors changes on your site’s first-impression (3-second stay rate) and the ability to build awareness (7 and 15-second stay rates) and sends you a notification if these metrics swing. For example when a bug is deployed or if some new design made a noticeable difference.

Let’s look at the benefits

1. It’s all automatic. Receive notifications when an update is detected, when the short-term results are available, and when all traction figures reach statistical significance.
2. Develop like before without thinking of A/B testing. Use the deployment system of your choice (like Git), make updates when you want, and be sure that you get noticed if you do something bad.
3. Get results immediately. Volument uses leading indicators like the 7-second stay rate to measure the performance. These metrics are actionable, meaning that your edits have a direct impact on them. When you make a new design, you’ll know the impact in hours/days instead of months. For every bottom-funnel conversion, there’s tons of data at the top.
4. Know everything from the update: how it impacted viewport consumption, retention, and conversion build-ups and how the funnel performance changed from top to bottom.
5. No need for crazy stuff like bayesian math, or pre-validation with A/A testing. Website optimization should not be rocket science.
6. Start building something people want. When the website development moves away from marketers to developers and designers — the website is clearly in better hands. It looks better and works better. There’s less bloat and tracking scripts. And no ugly FOOC effects from the visual A/B testing tools. Remember: the whole idea is to increase the number of conversions, right?

Automated A/B testing is introduced later this year.

Privacy becomes a norm

The use of privacy-friendly analytics is only going to be on the rise and might even become a standard in the near future. Two reasons:

1. Online identities are not needed for marketing analytics. All the layers on the conversion funnel are fully measurable without knowing peoples’ identities. Identities matter only at the very bottom of the funnel. They are essential for CRM software and the advertising industry, but not for conversion optimization.
2. People hate the cookie banner. The way it’s implemented today is a dark pattern: you are gamed to “accept all cookies” which allows the site to track/store/share your behavior without limits. Websites that respect your privacy are more trustworthy and thus become more popular.