Mike Sharkey, our guest blogger, is the CEO of Autopilot, easy-to-use software for multi-channel marketing automation. Check out the Autopilot blog to learn more about marketing automation, lead nurturing, and creating remarkable customer journeys.


I had just finished watching 10 videos on UserTesting.com. It was late in the afternoon and I hadn’t slept in days since we launched Autopilot. I was tired and stumped. Since launch, our test users were nailing our onboarding flow 9 out of 10 times for a killer activation rate. But for some strange reason real users were having a hard time with the exact same flow.

Our UI designer, a few software engineers and I just couldn’t figure it out where users were getting stuck. We watched video after video analyzing every twist and turn in the onboarding experience, sometimes patting ourselves on the back and other times disappointed because a tweak we made didn’t work like we planned. We felt like we were missing something. Then out of nowhere came a perfectly-timed email.

Enter FullStory.

At first I didn’t believe FullStory would work for our app. Watching replays of user behavior on our website would help with marketing, sure, but I was more interested in helping customers onboard and teaching them how to set up marketing automation in a few clicks. I wanted to increase our activation rates and make users fall in love with our product, quickly.

Despite my hesitation we added the simple JavaScript code to our website and onboarding process. Then we waited. Moments later and like magic I could see user behavior from the moment a new user signed up all the way through our onboarding process. I was hooked.

To my surprise, watching the onboarding process from the perspective of real users (instead of paid test users) was radically different. We were able to get true behavior insight and avoid the much dreaded Hawthorne Effect typically associated with user experience studies. Seeing real user’s roadblocks made us focus on the what, why, and how of the product.

Tracking Metrics.

To give context, Autopilot is a product that helps companies create automated marketing flows (we call them “journeys”) with email, text, and direct mail. Like other SaaS companies, we’re laser-focused on acquisition, activation, retention, referrals and revenue.

Increasing our activation rate was top priority at this point. We defined “activation” as a user publishing at least one journey in Autopilot.

We used Segment and Mixpanel to track when users signed up and published their first journey. Using these two key metrics, we built a funnel in Mixpanel to observe our progress.

Mixpanel funnel early in our Beta.

We A/B tested everything in our onboarding experience. As we made changes we would look at the effect on our simple funnel in search of the modification that would skyrocket activation rates.

More visuals, less text.

In our paid user testing videos we observed the majority of participants would read each onboarding step out loud. Often they would misread the step, or make wild assumptions about what we were asking them to do to complete the step.

After using FullStory we noticed radically different behavior. No one read the steps at all. They had no motivation at that stage to read the steps. Users quickly skipped through often appearing to be confused or failing to complete the onboarding tutorial.

To improve this, we significantly reduced the text in each step and focused on simple images showing the action required to complete each step.

0-2Sh5XpQjkBzo6QTy

An early example of an updated step using images instead of text.

Frantic clickers.

We also learned getting people to focus on one thing isn’t easy. People frantically clicked around trying to complete each onboarding step. They ended up accidentally exiting or incorrectly completing the step, making the process confusing and destroying activation rates.

To solve this our engineers and designers developed a way of disabling our user interface to only allow the user to do two things:

  1. Complete the step; or
  2. Exit the walkthrough.

Active click or drag areas are limited to help the user focus on the step

By implementing this change we observed an overall increase in our activation rate, from an average of 30% (before the change) to 42% after the change.

A-ha or bust!

After four weeks of metrics we could see a trend towards those who published their first journey and those that hadn’t. 55% of those that did publish, purchased a subscription to Autopilot.

The problem was our onboarding tutorials (which have an average completion rate of 42% at the time of writing) didn’t focus on helping a user publish a journey. Instead, they focused on helping the user get Autopilot setup and configured while learning the basics.

Using FullStory we could see many users who signed up for a trial would simply complete the onboarding, click around and then leave. We surveyed a small sample of those that completed the onboarding tutorial but didn’t publish their first journey. We also contacted many of these people to learn directly what the problem was, and learned very quickly we were failing to address the why and what of marketing automation.

Marketing automation is still a relatively new category with low adoption (less than 4% in the US alone) so convincing users why marketing automation is valuable and what types of journeys to create is critical. Put simply, we were missing the “Aha!” moment.

A whole new onboarding experience.

To help users reach their “Aha!” moment, we redesigned our onboarding tutorial from the ground up. This time we focused on (with less clicks) helping a user publish a journey that sends two emails based on a time delay and an SMS using them as the test user. The only work they needed to do was to add the second email and click publish. After publishing the journey they experienced what it might be like to send customers automated communications on different channels.

Our new onboarding focuses on less clicks and publishing a journey.

From working with customers we know the “Aha!” moment in Autopilot is when you publish your first journey and start seeing responses. This whole new onboarding experience is going live soon, and it’s our hope it will have a dramatic impact on our activation rate.

The results so far.

For the sake of transparency, here’s our most recent trial funnel metrics:

This is a 15% higher activation rate than during our beta. But, it is still too early for before and after metrics with the new onboarding flow. I’ll follow up this post with our continued learning and results once we collect more data.

Using a combination of UserTesting.com and FullStory has allowed us to test assumptions, observe behavior and set goals to improve our key metrics. UserTesting radically improved the performance of our onboarding experience and helped us remove the kinks, but FullStory gave us insight like we never had before — the ability to observe the behavior of users in the wild.