How accurate are privacy-focused analytics platforms? [30 days experiment]

Published on Mar 19, 2020 in Business


Privacy-focused analytics platforms are becoming increasingly popular. After Paul Jarvis led the way with Fathom, many others followed suit.

These tools promise to respect your visitors’ privacy by not using cookies nor collecting personal information, while at the same time offering a simpler user experience for site owners.

In this respect, the difference is stark.
Unlike Google Analytics, which offers dozens upon dozens of different types of reports and infinite ways of segmenting the data, these tools show everything on a single page, making GA look like a clunky relic from the 90's.

But just how accurate are these tools compared to Google Analytics?

If you are seriously considering switching to one of these GA alternatives you want to be sure you can trust the data. Privacy or not, when it comes to analytics, data-accuracy comes first.

So a month ago I decided to do an experiment and see how accurate these tools actually are. I signed up to 3 tools and installed their script on one of my websites.

These are the three tools I decided to test:

  • Fathom
    Fathom is the category creator and probably the most popular privacy-focused analytics tool. It was created by Paul Jarvis and Jack Ellis.
  • Simple Analytics
    Simple Analytics was launched quite soon after Fathom by Dutch indie-maker Adriaan Van Rossum and it's very popular amongst indie makers like Marc Köhlbrugge and Pieter Levels.
  • Metrical
    Metrical is one the last players in the market and it's officially still in beta, but you can request access. It was created by my Italian friend Francesco Marassi

In this post I compare the analytics collected by these 3 tools against those collected by Google Analytics from the 18th of February to the 17th of March 2020. Google Analytics, while not perfect (many ad-blockers block GA), will be considered the benchmark of accuracy.

Hopefully this post will help you decide if you should use a privacy-focused analytics tool and which one.

Metrics

Since privacy-focused tools offer on a handful of metrics, I will only compare those. In particular, the metrics I’ll focus on are:

  • page views
  • unique visitors
  • avg time on site
  • top page visited
  • top referrers
  • goal completions
     

Page Views

Software Page Views

Google Analytics

6,445

Fathom

6.7K (the actual number is not shown)

Simple Analytics

6.1K (the actual number is not shown)

Metrical

6,451

Page views is one of the easiest metrics to track because you simply need to record a hit every time the script is loaded. Despite this, Fathom and Simple Analytics are slightly off by about 200-300 page views while Metrical is very close to Google Analytics. 


Unique Visitors


Software

Unique Visitors

Google Analytics

2,150

Fathom

2.7K (the actual number is not shown)

Simple Analytics

2,865

Metrical

1,446

This is where things get interesting. Without cookies, tracking unique visitors becomes really hard to track and each tool handles this in a VERY different way.

Fathom uses a system of “multiple, un-related complex hashes” that is frankly beyond my pay grade and is explained in detail here.

Simple Analytics detects a unique visit based on the hostname of the referrer of the page. From their docs site:

"If a user comes from one domain to another it shares the previous domain with the next via a so called referrer. If the domain is the same as the one in the referrer we know it’s a non-unique visit."

I'm not convinced by this approach. A person who comes from a mobile app or an email client doesn't carry a referrer. The same person can click on the same link twice and he or she will count as two uniques.

Metrical uses yet a different approach, loosely based on Fathom’s, that involves creating a complex hash (session ID) using some user data (IP address, User Agent, the day of the year, etc) and then check whether or not another Session ID with the same value already exists in the database.

Nonetheless, all three tools are quite far off the number recorded by Google Analytics. The "most accurate" is Fathom despite having recorded ~550 more unique visitors (25% higher!)

This is perhaps where privacy-focused tools are at the biggest disadvantage compared to a cookie-based analytics platform.


Average time on site

Software

Avg Time on Site

Google Analytics

2:04 min

Fathom

1:19 min

Simple Analytics

Not recorded

Metrical

3:24

Average time on site is historically one of the most difficult metrics to track and this is proved here as well with Fathom and Metrical differing wildly from the value shown by GA (-37% and +65% respectively).

One word of caution here though: for Google Analytics the avg site is calculated by the time difference between the point when a person lands on the page and when they move on to the next one. If the person exits the website without going to any other page, then the time-on-page is zero.

On the other hand, Metrical uses window.onfocus to detect when the tab is on focus and the visitor (presumably) is actually looking at it. I couldn’t find any information about how Fathom calculates avg time on site.

Frankly, I don't think this matters a lot. Avg time on site is not a very useful metric anyway, especially for the type of people who use privacy-focused analytics platforms (more on this later).

PS: Adriaan of Simple Analytics told me that he will add time on page soon.


Top pages visited

Software

Top 4 Page Visited

Google Analytics
  • Homepage - 2,220 page views
  • /features - 791 page views
  • /pricing - 766 page views
  • /examples - 511 page views
Fathom
  • Homepage - 2,300 pageviews
  • /features - 867 page views
  • /pricing - 844 page views
  • /examples - 543 page views

Simple Analytics

  • Homepage - 2,500 pageviews
  • /features - 765 page views
  • /pricing - 711 page views
  • /examples - 447 page views

Metrical

  • Homepage - 3,000 pageviews
  • /features - 824 page views
  • /pricing - 762 page views
  • /examples - 468 page views

No big surprises here. All three tools have the same 4 pages as the top pages visited. 

Top referrers

Software

Top 4 Referrers

Google Analytics
  • Google - 702 unique visitors
  • Indie Hackers - 181 unique visitors
  • Product Hunt - 45 unique visitors
  • Twitter - 37 unique visitors
Fathom
  • Google - 1.1K unique visitors
  • Indie Hackers - 253 unique visitors
  • manuel.friger.io - 204 unique visitors
  • Product Hunt - 72 unique visitors

Simple Analytics

  • Google - 906 unique visitors
  • Indie Hackers - 210 unique visitors
  • manuel.friger.io - 155 unique visitors
  • emails - 41 unique visitors

Metrical

  • Google - 410 unique visitors
  • Indie Hackers - 105 unique visitors
  • manuel.friger.io - 73 unique visitors
  • Product Hunt - 28 unique visitors

Things get a bit weird here. All three privacy-focused tools show manuel.friger.io (this website) as a referral which is nowhere to be found on GA. I honestly can't explain it.

Other than that, the differences are minor. Simple Analytics seems to have slightly more similar results (to GA) than the other two. Some trends are also starting to emerge; for example Fathom tends to consistently show higher page views and uniques than GA, while Metrical tends to show consistently lower figures.

All tools group referrals by domain and allow you to click on each referral to see the exact URL of the pages your visitors come from, however Metrical categorises sources using the traditional three categories - referrals, organic and direct, which I personally prefer.


Goal completions

Software

Conversions Tracked

Google Analytics

137

Fathom

156

Simple Analytics

Not recorded

Metrical

159

Any analytics tool worth its salt must have a way to track custom events (e.g: signups, purchases, clicks, etc) and all three tools have this feature.

Unfortunately during my test Simple Analytics was undergoing an update which prevented me from using this feature. Event tracking on Simple Analytics is currently described as "highly experimental".

Fathom and Metrical have had event tracking in production for some time but it’s still quite basic. They only allow you to track events by calling a Javascript snippet and events are mono-dimensional in the sense that only the event name is passed on. Unlike Google Analytics, events can't be categorised and grouped.

Both Fathom and Metrical have recorded a slightly higher number of conversions. There are two possible explanations here. One is that since these tools struggle to record unique visitors properly, sometimes two events triggered by the same person (which should be recorded only once) are recorded twice. Another explanation is that Google Analytics is sometimes blocked by ad-blockers and thus it has recorded a lower number.

Bits and bots

A meta-analysis wouldn't be complete if I didn't consider other aspects/features of these products.

Extra metrics

All three tools show stats about Device Type (phone, tablet, desktop), Browser (Chrome, Safari, etc) and Countries but Metrical also shows the language of your visitors. Fathom is also the only tool that shows the bounce rate of your website, though a global value is not nearly as useful as the bounce rate of specific pages.




Filtering

Metrical is the only tool that allows to filter the data. For example, I can click on “desktop” and all the other stats (page views, unique visitors, top pages, etc) will be filtered with only visits that came from desktops. I can even use multiple filters at the same time. For example, if I want to know the top sources of people visiting from the United States using a desktop, I can simply click on “Desktop” in the “Device Type” section and “United States” in the “Country” section.




Real-time

This is probably something only a tiny minority of people are interested in but both Fathom and Metrical offer a real-time view, while Simple Analytics doesn't.

Custom domain

One of the biggest problems of analytics scripts is that they are blocked by ad-blockers. A genius solution, offered by both Simple Analytics and Metrical, is to use a domain bypass, which means the script doesn't use Simple Analytics' or Metrical's domains and thus ad-blockers won't prevent the script from loading. 

This feels like such a cool and useful feature that I'd be surprised if Fathom doesn't add it soon.

Export data

For the data geeks among you, both Simple Analytics and Metrical allow you to export all your data while this doesn't seem to be possible with Fathom.

Pricing

After 15+ years of using GA for free we might have forgotten a simple truth: analytics cost money and privacy comes at a premium. Thus, Fathom, Simple Analytics and Metrical require a monthly subscription to be used.

Fathom starts at $14/month for up to 100K page views/ month. For the same, Simple Analytics costs $19/month. Metrical is currently in beta but the founder told me he's going to charge $5-$7/month.

Are privacy-focused analytics platforms for you?

Compared to Google Analytics, privacy-focused tools are about 10%-20% less accurate, on average.

If you have a blog, a personal website or a small business, this is probably acceptable and I'd definitely recommend one of these tools over Google Analytics. They give you 90% of what you need with a much better and simpler user interface.

However, if you have an e-commerce website, a SaaS website or simply a large website that requires accurate data, custom segmentation and advanced event tracking, Google Analytics is still your best choice.

Privacy-conscious customers are on the rise and privacy-focused analytics platforms are definitely here to stay.

But after using these tools for a month and chatting with several people about them (including the founders), I've come to the conclusion that the real value proposition of these tools is about the ease of use and simplicity while privacy is a (nice) bonus.

I don't dispute the motivations of the founders of these products. I genuinely believe they are motivated by noble privacy-related ideals, but I think most of their customers are primarily drawn to the simplicity of their platforms (compared to Google Analytics) and helping small, indie businesses.

And the winner is...

Personally, I liked all three tools but my favourite was Metrical. Here's why:

  • Metrical is a nice mix of Fathom's unique features (real-time, avg time on site) and Simple Analytics' unique features (custom domain, data export) and also is the only tool to offer filtering.
  • In terms of accuracy, whilst not the most accurate (Fathom probably wins here) it has consistency erred on the conservative side
  • I prefer Metrical's categorisation of sources over the other two
  • Metrical will cost half what the cheapest alternative (Fathom) costs.

For full disclosure I must say that I know Metrical’s founder, Francesco, personally but this is my objective opinion. Metrical has the highest number of features, is accurate enough and is cheaper than the alternatives which makes it my favourite choice.


Share "How accurate are privacy-focused analytics platforms? [30 days experiment]"

Manuel Follow me on Twitter