How even the most tantalizingly beautiful data visualizations can go completely wrong
Wouldn't it be incredible if you had a way to quickly see how your aggregate user base engaged with your website or app? Well, heat mapping tools claim they can do just that.
But can heat maps deliver the goods? And at what cost to the reliability, accuracy, and actionability of the information? What are heat maps, really, anyway?
What are heat maps?
Heat maps are a data visualization that aggregates user behaviors on a website or app using red, yellow, orange, green, etc. semi-transparently overlaid to the screen being analyzed. Heat maps vary color intensities to convey the intensity of user interactions, attention, or eye-movement.
Visually, heatmaps can take many forms:
While heat maps are simple enough in theory, in practice they are complex.
Consider these challenges.
How do analytics tools even know where users are paying attention, clicking, or mousing over? Here again, there's no common approach. Heatmaps can determine areas of user interaction through eye-tracking tools, session replay tools, or perhaps just aggregated clicks on links.
While heatmaps have become the go-to graphic for showing user interactions on a web application, they are far from perfect. And perhaps its not surprising that when it comes to heatmaps, the devil's in the details.
When is a heat map not a heat map? The history
Let's back up. The term “heat map” or “heatmap” was trademarked by Cormac Kinney in the 90s to refer to a 2D financial market visualization (Wikipedia; also Kinney’s lapsed Trademark). Today, what most of us think of as heat maps—as evidenced by our simple definiton and sample heatmaps above—is entirely different.
According to Kenneth Field (a.k.a. Cartonerd) in his post When is a heat map not a heatmap, most of the data visualizations referred to as “heat maps” are not actually heat maps. They are technically isarithmic density maps.
What is an isarithmic density map? We'll get there, but first it's useful to take out the "density" and speak to just isarthmic maps, which the Geographic Information Systems encyclopedia defines as:
A type of thematic map that represents a continuous field using isolines (lines of equal value) or isopleths (regions of similar value). They are sometimes referred to as a heatmap, although a heatmap is only one type of isarithmic map that represents density.
A popular form of isarithmic map is a contour line map. Think about topographic maps — each connected line in a topographic map represents the same elevation:
Your run-of-the-mill topographic map uses bright line data points — elevation at specific, unique geographic locations — and interpolates (i.e. connects the dots) the missing information across the data points.
That brings us to web application isarithmic density maps—a.k.a. the things we all think of as heatmaps. Web heatmaps don’t have isolines. Instead web heatmaps use isopleths. Isopleths are used to represent regions of similar value — red, orange, and yellow, green and blue color splotches.
The splotches of color on heatmaps represent aggregated user experiences on a web site or application. As data visualizations go, heatmaps make for an easy-to-grok, pretty picture. You look at heatmap analyses and say, “Ah, I see!” You feel like you understand something deep and meaningful—like you're peeking behind the curtain and seeing a remarkable bit of web analytics.
But there’s a catch — or at least four catches — and if you’re hypnotized by the glowing hotness of heatmaps, you very well might miss these problems completely.
4 Major Problems With Website Heatmaps
As we see it, there are at least 4 major problems with heat maps—these are showstopping obstacles because they call into question the very reliability of the data. See if you don't agree.
1. Lies, damned lies, and [X, Y] positions
In a world of responsive design and device heterogeneity, using a visualization designed for a fixed unchanging aspect ratio is a recipe for failure. “100px down and 350px to the right” doesn’t mean the same thing across two different user visits to your site with each at different browser window sizes and/or screen resolutions.
Ignore these differences and you might find yourself at a confusing dead end. Responsive design and varying display PPIs also make converting between aspect ratios only possible with a pile of assumptions.
Heat mapping tools get around this problem by forcibly siloing the datasets they visualize. You only see heatmaps with aggregate data for users with the exact same screen resolution configuration. 1024×768 users go here. 1366×768 go there. 1280×800 users go in the cupboard. These arbitrary silos mean you aren’t actually seeing the meaningful aggregate across your actual non-same-screen-resolution-using users. Kinda defeats the purpose of the “easy” visualization, right?
And this is just on desktop. There are thousands of different mobile browser resolutions in the market. Most heatmap visualization solutions pretend these differences don’t matter.
Yet they matter a lot.
Averaged and aggregated mouse hovers and un-contextualized [X, Y] click positions cause well-intentioned product teams to make actual product changes based on shoddy information, wasting time and money, negatively affecting the business, and harming customer experience.
2. It’s 2018: Dynamic applications are the norm
Today, a single web page is no longer a static fixed document. Modern webpages and applications are dynamic. Animations, slide out panels, expanding menus and modal dialogs are the norm. As you might guess, this poses a problem for heatmaps.
You end up with aggregated positional spaghetti.
And this same issue affects everything from product inventory tables on an e-commerce site to data-driven menu options in your business enterprise app.
3. Splotch analysis is botched analysis
Even when the stars align and they provide accurate and meaningful aggregations of user interactions, the heatmap visualization itself doesn’t directly tell you what you want to know. Those aggregated splotches obscure the very thing that was interacted with. You have to read the heatmap tea leaves and interpret the splotches to figure out what you think the users were interacting with. What button, link, or control does that blob correspond to?
You might call this effort “splotch to element mapping:”
After seeing a heatmap and completing your back-of-the-envelope “splotch mappings,” what do you do next?
Here, heatmaps quickly turn into "analytics theater." Analytics theater is when an analysis—via a graphical visualization, chart, graph, metric, ect.—dazzles your senses with signals indicating profound insights while in reality being either completely useless or so difficult to parse into action as to become useless.
Heatmaps are notoriously hard to convert into actionable business intelligence ("actionable insights"), and if you can't make clear use of the insights, heatmaps become little more than beautiful distractions—analytics theater.
4. Did we mention the configuration?
Most heatmap solutions ask you to configure complicated URL patterns and regular expressions to tell the system how to interpret interactions on a page-by-page level.
If heatmaps are broken, how do you visualize aggregated user interactions on your site?
Click Maps—Heat Maps Evolved
Having thoroughly examined heatmaps as a possible solution to meet our customer’s requests for a visualization of user clicks on a web application or webpage, we decided heatmaps needed to evolve into something better.
That's why we built Click Maps to make it easy to see how your users are interacting with your website, in aggregate, while gracefully dodging the four major problems of heatmaps.
Page Insights are resolution independent
FullStory understands the structure of your site. FullStory can translate that into aggregations that make sense across screen resolution, device type, and even the most dynamic and varied data driven interfaces by focusing on Elements. Not just position.
Page Insights take into account what’s on (or not on) the page
If you watch a playback of a dynamic single page application on FullStory, you realize that many things pop in and out of existence.
Page Insights are smart enough to account for these dynamic states and show you what’s relevant to what you are seeing on the page now. Click analytics for a non-existent popup won’t be shown when the popup isn’t there.
Page Insights are actionable
You can click and see your answers immediately. And if you have questions you want to drill into, you can search. No orange splotch mapping to your web app required.
See here for an example of how you can run analysis using Click Maps.
Page Insights require zero configuration
FullStory uses machine learning to know what the pages are in your web site or web application. FullStory understands what pages are the same, and what pages are different. So your insights are appropriately relevant and contextualized. No need to do complicated configuration or setup brittle URL route regular expression voodoo. It just works out of the box.
Bonus! Some new tricks
Page Insights give you aggregations that respect the searches you do in FullStory, too (If you’re not familiar, here’s an explanation of FullStory search). This helps you answer questions like “What do visitors from my reddit ad campaign click on first?” Or you can see how they compare to “visitors that came from Product Hunt.”
Page Insights make your existing FullStory segments even more useful.
See ya later, web app heatmaps! As for the default data visualization for web app interaction of yesteryear — that is, isarithmic density maps … er … heatmaps — until the day comes when we can reasonably manage the problems they create, we’re leaving heatmaps where they left us: out in the cold. And if you want to try visualizing aggregate customer experience on your web application, check out FullStory Click Maps, see what you can discover and take action on, and let us know how we can make it even better.
🔥 FullStory Update, Summer 2018 🔥
Click Maps get a friend: Engagement Heatmaps 🍦
Despite the reservations we shared a year and a half ago, we always wondered if we could offer up the "easy-to-grok, pretty picture" you get with heatmaps and navigate the four major problems we saw with existing, traditional heatmaps. But these things are hard and take time.
Today, we're excited to share we've created a new way to see aggregate engagement on your website at a glance—while maintaining the clarity and precision you need to act.
Meet Engagement Heatmaps. These heatmaps are so hot they're cool: a Superman ice cream-inspired way to visualize engagement on your site or app. Engagement Heatmaps preserve all the accuracy of Click Maps while requiring less cognitive load to understand. Intrigued and want to know more?