You finally launched your shiny new analytics tool—Ultimate Analytics ProTM. It's going to be great.
It better be—this analytics platform is expensive and you sold it through to stakeholders across your org. The expectations are high.
Firing up the tool for the first time you're greeted with a dynamic dashboard filled with fancy bar graphs, provocative trend lines, tasty pie charts, and loads of user metrics. It's all so much to take in but you feel good. You rest assured that you'll finally have the business intelligence you need to help your organization make better decisions.
Yet as you circulate your fancy new reports, the days turn to weeks and you get a nagging suspicion that very little has changed.
Something isn't working. But what is it?
Security theater and signaling
A little over ten years ago, security expert Bruce Schneier wrote Beyond Fear and coined the phrase "security theater." Security theater is when an organization engages in behaviors that signal security even if those activities don't provide any additional, real security, at all.
The most commonly cited examples of security theater in recent years tend to involve the TSA—that is, airport security—but if you've ever noticed the blinking red light in your locked car you've witnessed a bit of security theater.
Security theater is a form of signaling. What's signaling? Signaling is when information is communicated indirectly through a medium separate from the underlying reality. E.g. gestures, actions, products, behaviors, etc.
Signaling is everywhere—from banks being made of marble, signaling wealth and stability, to little puffer fish creating "art" on the ocean floor to signal fitness to potential mates.
You might think of signaling as marketing or advertising—trying to convey a distilled, simple message about something more complex. When signaling works, the signals are accurate representations of the underlying reality.
When signals don't work, they take on a life of their own, regardless of their purpose. Security theater is an example of signaling for its own sake—sending a message of safety that doesn't hold up under scrutiny.
And when it comes to theater, while we might be convinced by the show, it's little more than fiction. Can you spot the difference?
Signaling and Analytics Theater
The purpose of analysis is to distill complex underlying realities into salient insights—the kind that inform decisions. Just like signals. The signals in analyses come in the form of visualizations (charts, graphs, etc.) and metrics. Consider this trend line:
Without knowing anything about the data behind the chart, what's the signal? You might guess that this graph represents an underlying reality that is good. If you saw this kind of graph in the context of a presentation—accompanied by a narrative you have no reason to question—you'd likely see it and nod your head in understanding. After all, that's what up and to the right means—warm fuzzies.
Look at the following trend line and you'll get a very different, yet equally clear visual story:
Ouch. That doesn't look good. Someone has some 'splainin' to do!
What's happening? As discerning as we may be, when you run into enough analytics, you start getting trained on their conclusions. You process them automatically. Indeed, the power of signals is that they can tell us something without us having to think critically.
That lack of critical thinking can be a problem. The decisions we make based on signals have real effects. We need to exercise caution and do less mindless headnodding, which is why the next time you see some warm fuzzy-generating chart—or wince-inducing graph, let that be a different kind of signal—one to start asking critical questions:
- Does the graph accurately represent the underlying reality?
- Is the "insight" you're getting telling you something useful?
- Am I better able to do my job because of this analysis?
If you answer the above questions, "No," your analytics signals may have gone off the rails—you may be experiencing analytics theater.
The fact is the savior, as long as you don’t jam it into some preconceived pattern. The greatest obstacle to discovery is not ignorance—it is the illusion of knowledge.
Analytics theater is when your analytics tools provide you with signals—metrics, charts, graphs, whatever—that give the appearance of valuable insights while being mostly if not completely useless and/or poorly reflecting the underlying reality.
We've all experienced analytics theater. Very likely, we've even produced a show or two. Where are you most likely to find analytics theater?
- Automatically generated dashboards
- Monthly reports (usually trotting out the same metrics)
- Ad-hoc analyses used to justify management decisions
- Powerpoint presentations with slides and slides of graphs and numbers
Mind that none of the above instances need necessarily be analytics theater. There are plenty of useful reports and dashboards.
However, next time you see that fancy dashboard with all its metrics and graphic visualizations, ask questions. Are the analytics signaling something insightful or simply summarizing performance? Having read through the "insights" in your report, do you feel compelled to take specific action?
Heatmaps are a beloved form of visualization popular in web analytics. Heatmaps highlight in aggregate where users most frequently click, mouseover, or pay attention (depending on the heatmap tool). But there are a lot of problems with heatmaps, a topic we have explored at some length.
When it comes to analytics theater, just as much as heatmaps signal UX insights through their unique depictions of user behaviors, they are difficult to parse into actionable insights
Useful analytics have actionable insights
If you want to know if the signals you get from your analytics are more than analytics theater, you have to pay attention to outcomes. Consider:
- When you get a signal from your analytics tool, do you have an idea of what you should do next?
- Does a visualization drive you to clear next steps?
- Does a metric tell you enough to deduce what is happening?
- Having absorbed it, do you end up more capable of doing your job ?
Useful analytics signals contain insights that drive you to action. And we don't mean the buzz-wordy kind of "drive," either. We mean the "We need to do this now" kind of "drive." That is, "The kitchen is on fire and we need to extinguish it now," or "The radio station is giving away front-row concert tickets to the 17th caller—where's my phone?! Now."
Real analytics drive action, and the best analytics tools produce loads of insights you can put to use.
Expect more from your analytics tools
A little bit analytics theater is harmless and unavoidable. Who doesn't like the occasional warm fuzzy—like seeing DAUs (or revenue!) go up and to the right? And sometimes a "status update" is all you need.
Just don't confuse those signals for the kind of insights on which you can take action. Remember, you've spent a lot of time and energy on your analytics platform, so it better be giving you more than status updates.
If you really want to put your analytics to work, demand that your analyses spit out more than pretty charts and graphs: make sure it provides you with analyses that are useful, actionable.
Otherwise, you're just putting on—or watching—a show.
We might have a suggestion for just such a tool. ↩︎