How to take advantage of both quantitative analytics and qualitative research
Understanding how people interact with your website, app, or digital product is hard. You can have all the user data in the world but no idea what to do with it, no process to manage it, and no way to filter the signal from the noise.
Where do you even start? And once you begin, where do you go?
“Data saturation is everywhere. We’ve often had the belief that more is better; however, that actually isn’t true in the case of data. The rapid rise in our ability to collect data hasn’t been matched by our ability to support, filter and manage the data. [We have] too much data with not enough structure in place to manage the data and not enough meaningful application.”
— Sara Spivey, CMO at Bazaarvoice (Forbes)
What you need is a systemized approach—and the right tools.
When it comes to knowing, traditionally there have been two approaches you can take. The first is the quantitative approach. Quantitative analytics puts everything in terms of numbers. Number of users doing [whatever]. Bounce rates. Conversion rates. Time on site. That kinda thing. Quantitative research is a squeaky clean way to crunch numbers and understand what’s going on at a high level.
The drawback to quantitative data is that much can be lost in translation. Important nuances missed and in that absence of information, the quantitative can play tricks on our pattern-recognizing brains. We can see things that just aren't there—just like how we can see faces in objects.
To really know what happens on your website, app, or web application, you need a way to get at that nuance. And that's where the second approach comes in—the qualitative. Qualitative analysis focuses on the subjective qualities of your users—the nuanced actions that drive the numbers. It's descriptive research. The qualitative approach is subjective, too. The user paused here. Raged there. They seem confused with this UI—what's going on? Qualitative data is messy. But it is rich in insights.
The major problem with qualitative research methods is that they occur at the individual level. Qualitatively researching every user who visits your site is impossible. You could never do it, nor would it be useful.
The best analyses combine quantitative and qualitative research to create a flywheel of continuous product improvement
So what do you do? Use both.
The best analyses combine quantitative and qualitative research to create a flywheel of continuous product improvement. You use quantitative information to focus attention on the biggest problems (and opportunities) and then "zoom in" to the level of the user through qualitative research to round out your understanding—and solve the problem.
This one-two research method spins out insights you can turn into actions. It's a quantitative-qualitative flywheel that can be used for continuous improvement of your web site or app.
Here's the 3-step process for how it works.
1. Quantitative Data Alerts You to Problems
The most basic web analytics tools available are quantitative—and free. It comes as no surprise that one of the first things web site owners do on launching a new site is install Google Analytics (GA).
Traditional quantitative analytics tools like GA do a reliable job providing high-level metrics—e.g. user info, page visits, events, conversion rates, bounce rates, time on site, etc. Most importantly, they allow you to see the status of different aspects of your site and help you focus on problems or opportunities in aggregate.
With a functional quantitative analytics tool up and running, you're ready to identify issues and opportunities for improvement. This is a great place to put in motion a quantitative-qualitative flywheel—though as you'll see in this article, once you have the flywheel going, it's not the only place to begin.
When you have a quantitative tool tracking your customer data, it is easy to be dazzled by the illusion of insights. Fight the tendency to be hypnotized by the show. Come at your newfound quantitative data powers with an objective to avoid becoming overwhelmed.
Consider these changes in quantitative metrics that left team's struggling to understand the why behind the quantitative what:
- An increase in customer churn seen by Wistia couldn't be explained or reproduced. Why were customers churning?
- A drop in conversions, like the car insurance company Metromile saw when users started quote requests on their site but then left them unfinished. Why weren’t users completing the form?
- Decreased usage of certain features, like the customer service software company Kayako saw on analyzing feature adoption. Why weren't customers trying out new features?
- Increased page bounces, seen by the translation service SpanishDict on on their homepage. What was breaking at the moment of first engagement?
Remember: behind every quantitative metric is a real person trying to get some job done. You have to understand what your users and customers are trying to accomplish to smooth out the digital experience—and quantitative information can’t show you the nuance.
To figure out is what jobs they want to get done, you need to build out the next step in the flywheel.
2. Qualitative Data Gives You the Why
Having used quantitative analysis to identify problems that require additional analysis, you need a way to "zoom in" on individual user behavior. That's why you need a qualitative analytics tool.
Qualitative data can provide you with a ton of nuanced information about individual user behaviors. It's the most dense level of data. It's the kind of data you get with open-ended feedback—and also what you get through session replay. Using a qualitative analytics tool, you can see how individual users interact with your product.
Building a business case for qualitative analytics? See how the Product team at TrueCar leverages quantitative and qualitative analytics technologies together to increase engagement and conversions across their site and app.Get the Case Study
So how does it work?
How do you go from quantitative data to qualitative research? Using digital experience software like FullStory, you can search the entire catalog of user behaviors on your site for just those that would affect whatever quantitative data you're researching.
Continuing from our company examples above, qualitative data was used to uncover the why behind the change in quantitative data:
- Rage clicking indicated to Wistia that much of their customer churn was caused by confusion about how to use the product.
- Confused site navigation on Metromile's online dashboard showed the company that users didn't understand the value of their pay-per-mile model.
- Observing that most users skipped onboarding shed a light on why Kayako users didn't know about certain features or couldn't figure out how to use them.
- Seeing the site layout from the user's perspective helped SpanishDict see that users couldn't figure out how to get started because a banner ad was blocking the search bar.
In each of the above cases, qualitative research (completed using FullStory) provided the additional insights necessary to make sense of the problems identified through other quantitative tools.
Having used qualitative research to understand why you're seeing changes in quantitative data, it's time for action.
While the flywheel suggests using quantitative data to first identify problem areas within your digital experience and then digging deeper with qualitative data, you can also do the reverse. Many teams have a weekly get-together to watch session replays on FullStory, for example; these “gamefilm” meetings often highlight individual struggles within the digital experience.
If your qualitative analytics tool reveals a major point of friction for a single user, you can cross check that insight against your quantitative data to understand if that friction point is impacting many users.
3. Experiment and Check for Improvements
Armed with a complete view of the user from the individual to the aggregate, you now have the insight you need to do something, and that something is to make edits and run experiments. This is the last step in the quantitative-qualitative flywheel.
Just what edits are required will vary greatly depending on the problem you're solving. Make the edits, run the experiments (A/B testing is helpful here), and then measure your results.
You're back at the top of the flywheel, and on and on we go.
And You're Ready to Go Again ...
Rather than qualitative vs. quantitative, the most clever analysts use the two together.
Quantitatively identifying problems in aggregate and qualitatively understanding the underlying behaviors behind those problems has become a critical workflow (that, as you guessed, works in both directions!) that drives product improvements:
- For Wistia, user confusion about how to use the product was reduced through allocating additional resources to customer support.
- Metromile realized that they needed to add educational content to their homepage to reduce user confusion and encourage more quote requests.
- Kayako learned that they needed to overhaul their user onboarding, and designed a completely new onboarding flow to guide users through the part of the product they were most interested in.
- SpanishDict increased engagement by moving the ad banner that was blocking the search bar to optimize for mobile users.
The quantitative-qualitative flywheel ensures your brain will never have to scramble to fill in gaps. The best qualitative and quantitative analytics tools work together to reveal the full story of user behavior on your site.
Put the flywheel to work and reap the rewards of an improved digital customer experience.
What determines a functional analytics tool is a subject for another day, but no matter what, the analytics tools you choose should serve your needs and budget. ↩︎