FullStory captures a ton of data and uses this data to do lots of interesting things like power analytics, populate charts and graphs, create heatmap visualizations and, of course, render session replays. For many of our customers, FullStory’s session replay is their first experience gathering behavioral insights at scale and they dive in head first!
What tends to happen when people get FullStory up and running on their site is they quickly find themselves watching session after session after session with no end in sight. Before long, they ask one of the most common questions we hear at FullStory:
“How many FullStory sessions do I actually need to watch?”
This is a great question, and a really common one, especially coming from people who have less experience working with qualitative, behavioral data.
The good news is you can get a great deal of value from watching far fewer sessions than you might guess. Let’s explore how!
Follow the Rule of Five
Here’s the short answer: 5.
Watching five sessions per study (More on “studies” in a minute) will help you uncover roughly 80% of major usability problems. In addition, watching significantly more end-users will often fail to provide you with any additional insights and could very well be a giant waste of time. This is, in short, similar to the effect of diminishing marginal returns from economics.
As you can see in the graph above, as you add more and more user tests to your research you gather fewer and fewer additional new insights. You only need to see the same issue so many times before you are informed enough to make meaningful changes to your digital property.
Replay Five Sessions Per Study
Now, because a digital property has so much surface area, you can’t optimize the entire thing by watching only five session replays. You have to break your digital property up into smaller chunks of digital experience for this method to work properly.
(No, “chunks” is not a technical term.)
This is where “studies” come in. As defined here, a “study" has two main components. The first is the portion (think bite-sized) of your site or app you are looking to optimize—that is, the digital experience chunk. The second component is the group of end-users interacting with that piece of your digital property.
It is okay to go with your gut when determining how large the chunk of digital experience you want to study should be, but as a rule of thumb, I try to stick to specific experiences that take an end-user 10 minutes or less to complete.
Here are a few examples of these two study components for you to consider:
Digital Experience “Chunk"
- Homepage experience
- New user signup flow
- Checkout flow
- Newsletter opt-in
- Account creation
- Playlist creation
- Landing page experience
- First-time visitors
- Guest shoppers
- Mobile visitors
- Desktop visitors
- Repeat visitors
- Users from a specific region
- Visitors from paid clicks
You may scope your studies incorrectly a few times. Don’t worry. With some practice this should become quite intuitive. For end-user groups, just make sure that each session in the study comes from a unique visitor. You want to avoid all of your sessions being from the same end-user.
The portions of your site you choose to focus on first will depend on what kind of site (or app) you have, what your goals are, etc. Scoping your studies is the critical step to success with the Rule of Five. Meanwhile, the ability to scope a study based on whatever digital experience and/or end-user group you want is why, even while you only need to replay five sessions, you need all the sessions to ensure you can zero-in on what matters most for your research and analysis.
Thankfully, by testing your digital experience in bite-size pieces, you never have to worry about spending too much time investigating any single thing and you can deftly avoid getting stuck in “analysis paralysis.” Using the Rule of Five makes it easy to test multiple iterations of a page or flow in quick succession or to simply move on to investigating something new.
Let’s cap things off by looking at an example.
Example: Optimize the Checkout Flow
For an eCommerce company, there is arguably nothing more important than the checkout flow. As a result, this is often one of the first things people look to optimize using FullStory. In this example I will scope something a little more complex so you can get an idea of what it looks like to consider multiple user groups and devices.
Let’s get started.
My eComm company—we’ll call it CoffeeNet—sells mountains of beans across both desktop and mobile channels. Naturally, I am going to want to take this into account and watch sessions from both types of devices.
I also want to look at both first time shoppers and repeat shoppers to see if they experience the checkout flow differently. No UI should stand between my customers and their coffee!
Now, given these criteria, I would run four studies and watch five sessions for each of them:
Watching these 20 sessions will give me a good understanding of my major checkout-related usability issues. It will also provide insight into who, if anyone, struggles the most with this process. Now, be careful not to overthink your process for collecting insights. A simple spreadsheet will do.
After watching these 20 sessions, this is what I found:
Based on these results, it is obvious to me that first-time shoppers on mobile have more than their fair share of difficulties when navigating checkout. If I plan to spend time optimizing the checkout experience, starting there makes sense (assuming all other issues surfaced are of equivalent severity).
One last thing! After launching a new version of your checkout, it is a good idea to run another set of studies to validate your changes are working as intended.
That’s It—High-Five! 🙌
Replaying sessions in FullStory is an amazing source of behavioral insights just waiting to be uncovered—and you don’t have to watch thousands of sessions to get at them. (Although, if you want to watch thousands of sessions I won’t try to stop you!)
The next time you want to learn more about how your new feature, page, flow, or form is treating your end-users, remember:
Scope your study and just watch five.
Want to learn more about testing with five? Read NN/g's article Test with 5 Users.