How to prioritize user experience issues when a product feature isn't "Working as Designed."
Bugs happen. Software breaks. And whether it triggers an error notification or gets reported by a customer, bugs inspire valorous responses from engineers who rise up to squash and conquer.
Sometimes bugs aren’t so obvious. Apart from functional bugs—that is, bugs where the product is clearly broken—what about those more nuanced user experience issues that don't necessarily throw errors in the console? What do you do when you catch something—an issue with the user experience—that feels broken but might actually be "Working as Designed?"
It’s time to call a spade a spade. UX issues are bugs, and they should be treated as such. The only catch is that, unlike a functional bug, the "fix" for a UX bug is often undefined. With a functional bug, you have a design doc to reference the way it should work when it's fixed. With a UX bug, you need to go further to classify the problem, triage the work, and discover the solution.
That's where FullStory can help.
In this article, I'll walk you through a tried-and-true, repeatable process for handling digital user experience issues that's suitable for any product team. Whether you run a SaaS app or manage a critical part of an eCommerce site, you can follow these 6 steps to solve UX bugs:
- Start with Data—Instead of an Opinion
- Get the Bug in the Queue Where it Belongs
- Define the Desired Outcome
- Quantify the Impact
- Collaborate to Define the Fix
- Prioritize, Execute, Socialize
Grab your bug repellant and your head net, because we're going hunting.
1. Start with Data—Instead of an Opinion
If you've got FullStory in your stack, you're already ahead of the game in terms of finding UX bugs that need to be addressed. There are a few ways to back into UX bug discovery using FullStory.
Click Through from a Support Case
Your customer service agents might not be formally trained UX practitioners, but they've got killer instincts for ticket trends and the confusion that causes them. If you're starting your bug hunt from the support queue, you may be able to click through straight from a ticket to replay the session that caused a help desk request.
Look for the Rage Clicks
From the “Everyone” segment in FullStory (or any saved segment in FullStory) scroll down and look for the Rage Clicks. Rage Clicks reveal elements on which users are tap multiple times in quick succession (read more about Rage Clicks). Pick an element from this list of Rage Clicks and watch a related session.
Filter Sessions for a Critical User Path
If you don't already have a segment saved, use FullStory OmniSearch to build a quick funnel of sessions related to your most critical user path. For example, if your team is responsible for returning customers, you could spin up a quick segment for customers who click the “Sign In” button—for example, on your homepage—and then, upon entering their login credentials, click the “Continue” button:
In FullStory, watch for sessions related to the issue of interest and see what sticks out.
For the purposes of this blog post, let's imagine you found a session where a user tried to sign in as a returning customer. They land on the sign-in page, enter their username and password, and click "Continue" only to hit this error: "Error setting password."
Huh. That's funny. Why would a returning customer be setting a password when they're trying to sign in again? Wouldn't it be more likely that they entered the wrong password because they forgot or mistyped?
At this point, you have a piece of data that points to a UX bug. The form itself is functionally fine, but the error message on the returning user sign-in form doesn't match up with the expected customer experience.
Because the customer experience doesn't match customer expectations, classify this data point as a UX bug.
Now, while in FullStory, hit "pause" on the session at the time of the UX bug and click "Note and share" to open the annotation feature.
Try to describe the issue exactly as you see it, holding back any thoughts or feelings which could color the evidence with your take on the problem.
Bad note: Yikes! This is weird. Whoever wrote this error message totally goofed.
Good note: Customer sees error message "Error setting password" which is unexpected because they're logging in, not signing up.
Now that that's done, you've got a piece of data and a starting point for a conversation.
2. Get the Bug in the Queue Where it Belongs
If you have FullStory integrated with JIRA, that's gold. You can take the Note you just made in FullStory, click "Create Share Link," and then "Send to JIRA" to push the bug directly into your system.
Or if, like me, you need to follow a slightly more controlled process, here's what I recommend:
- Click "Report a bug" from your Note in the event stream or from the Notes page in FullStory.
- Click Copy to grab your note plus all of the other relevant details engineering might want for replication (viewport, browser, IP, etc.).
- Open a ticket in your system of choice (for me, that's Clubhouse) and paste the details from FullStory.
Now, before you launch your bug report into the ether, make sure it's:
- Titled clearly, using a naming pattern that will make it easy to recognize and find.
- Categorized as a bug (not a feature!).
- Assigned to the right group or project.
- Labeled or tagged appropriately, i.e. "ux-bug."
- In the right state for someone to triage and review.
Again, state the facts and lean on data as a starting point for a conversation. There will be plenty of time to address thoughts and feelings momentarily.
Hit whatever big button creates a ticket in your system and you're off!
3. Define the Desired Outcome
Now that the UX bug you caught is filed and logged in the system, it's time to take a step back. What is the desired outcome for returning users in this state?
Continuing with our example from above, returning users should be able to log in successfully and hit their profile page on our site.
So let's build a segment in FullStory to understand how many returning customers are able to do that.
Make sure to refine your search criteria to
in this order,
within the same session.
Then, scroll down to see the Event Funnel and learn what percentage of customers are able to make it all the way through to the desired outcome.
Save your segment and grab a snapshot of this funnel for later. You can even attach the screenshot to your bug report ticket.
4. Quantify the Impact
To precisely understand the impact of our example sign-in issue, we need to monitor our customer journey when the error was part of the customer digital experience. The most precise way to do this is by configuring Watched Elements to monitor who Has Seen the error message.
(If you've never used this feature before, you can work with a FullStory Hugger to set it up!)
With Watched Elements, you can build a copy of your funnel that specifically includes people who saw the error. If you don't have Watched Elements in FullStory enabled, you can refer to your standard funnel and look for the drop-off.
So with our example from above, let's quantify impact: How big is the drop-off between people who click "Continue" and actually reach the logged in state?
You can figure this out by first making a copy of your funnel. Then, flip the last step to
Has not visited (instead of who
Has visited) and grab the total number of people in that segment. Compare the two. This is the impact of errors within the sign-in flow for returning customers.
Grab that number and attach it to your ticket.
What's at stake in terms of your customer's journey? Again, continue with this scenario. If a returning customer doesn't overcome this error state, what critical action will they not be able to accomplish next?
Returning customers who can't understand the cause of a password error will not learn how to correct the inputs of the password field. If they can't correct the password field, they can't log in. If they cannot log in, they will either:
- Click "Forgot password?" and reset it
- Continue shopping and check out as a guest
- Bounce completely
Now, what are the potential costs of each of these alternatives in terms of both a) the customer experience and b) actual operational cost? Let’s take a look.
If they cannot log in, they will either:
Click "Forgot password?" and reset it
Experience Cost: moderately frustrating to the customer,
Operational Cost: $0.xx per reset email.
Continue shopping and check out as a guest
Experience Cost: mildly frustrating to the customer,
Operational Cost: measured KPI for $ value of checkouts from identified returning customers take a hit.
Experience Cost: mildly frustrating to the customer,
Operational Cost: lost opportunity of average cart size $0.xx.
Customer frustration comes with its own intangible cost. Each frustrated customer represents potential churn and/or lost word-of-mouth marketing. Don't forget to consider the real impact of frustration on the success of your business—even a rough estimate is useful here.
Also, you want to determine the total cost of any alternative paths for the customer.
Write these cost estimates down, and attach them to your bug report ticket.
5. Collaborate to Define the Fix
With a good sense of the opportunity at hand, rally the relevant humans to define the solution. In this example scenario, we're going to need at least:
- Someone who understands the code that's throwing the error (e.g. an Engineer or Developer).
- Someone who can design an alternative error state (e.g. a Designer and/or UX Copywriter).
Start asking questions:
- Technically, is the error that the customer is seeing the intended error based on the specific contents of the input field in this state?
- Based on the actual contents of the input field and the technical error behind the scenes, how would we make this error message more clear and specific?
- Given the scope of the issue, how much effort would be required to make this change? What is a reasonable amount of time that would be needed to design and code a fix?
Describe the solution (including any specific text or images as appropriate) and add all of the artifacts to your bug ticket in the system.
A note on exploring/suggesting solutions:
Designers are there to help you solve a problem. That's what they do. Usually, if you can describe the issue, they're going to come up with a solution that's better and more thoughtful than you could even imagine. If an issue gets stuck and isn't moving forward, it can be interesting to put forth a mocked up idea proposal. You can use a simple wireframing tool, PhotoShop, Google Slides, or even hack the UI and take a screenshot—whatever helps paint the picture. Sometimes a strong reaction against a less than great idea is exactly what you need to get momentum going for a better solution.
6. Prioritize, Execute, Socialize
Depending on the agility of your organization, it's possible that at this point, someone's already picked up your request and the fix is out the door. I love it when that happens, but don't count on it.
Is your ticket still in the triage queue? If yes, take a deep breath. It all comes down to this.
Identify the right prioritization process for your UX bug resolution. Is there a weekly meeting where bugs are reviewed? Is the effort big enough that you're going to need a Product Manager partner to help champion the work? Can you sneak it into the sprint for the next cycle?
Only you can know how to navigate the people and workflows in your organization. If all else fails, ask for help—preferably from someone who's been able to get their own "wish list" items prioritized before.
Finally, once your request gets prioritized for development, don't walk away! Like your golf swing, follow through matters with UX bugs. Once your fix goes live, be sure to revisit your FullStory segments and replay sessions so you can research and analyze results.
Circle back to your original funnel. Did you move the needle? Calculate the total impact and multiply that by the associated cost you defined. How'd you do?
Win or lose, this is your opportunity to share the story. Grab one of the original sessions that contained the UX bug. Quantify the results. Put together a narrative with feeling.
If you're not sure where to start, here's a little narrative template you can use to get the ball rolling:
On [this day], we caught a UX bug. The customer was trying to do [x] and they ran into [y] error so they weren't able to do [z].
Let's watch the session ... Wow! Painful, right?
In addition to causing customer pain, this cost us this many $$$ / this many #s for our KPIs.
We shipped a fix on [this day]. And after [this amount of time], we saw [these results] which saved us [this many $$$ / our KPI increase by this #].
Whether you share your story by dropping a summary into a Slack channel or presenting at the weekly all-hands meeting, make sure you get heard. When you're able to tell stories that connect your coworkers with your customer experience, those UX bugs stop feeling so much like "bugs" and more like opportunities to win as a team.
Yes, Try This At Home!
How does your organization file UX bugs? Have you tried this process in your organization? Drop us a line or give us a shout on your social media platform of choice. We'd love to hear your story.