Analytical data is a valuable tool for understanding user behavior on our websites. However, it’s the users’ emotional experience with your website — the ‘why’ behind their behavior — that reveals the truly valuable insights for marketers. This post will address why relying solely on analytics has its limitations, and show how qualitative research can help you bridge those gaps and create a personal connection with your users.

“Don’t make me think.”

Let’s start here, with this simple mantra. Or, more specifically: 

Don’t make users think.

We want to create experiences that are intuitive for users, not just performative. This is where qualitative data can play a crucial role. It can help reduce friction points, and lower the cognitive load for users (i.e. the amount of information our brain’s working memory needs to process). 

Both independent studies and the user testing we’ve undertaken at Kanopi have shown that when websites are intuitive, users actually perceive them to be faster. The reverse is true, also — when users are having a tough time navigating a website, they perceive it to be about 15% slower than it actually is. 

As a website owner, this is where these stats can really nail you: when users have a bad experience using a website, they tend to talk about it with friends and colleagues — and they recall it as 35% less performative than it actually was. 

Using qualitative data to create intuitive, non-friction web experiences are a great way to reduce user frustration across your entire website.

Quantitative Data vs. Quantitative Data

Data drives decision-making. At Kanopi, it helps us allocate budgets and resources towards our client projects. And it helps to think of these two distinct types of information as opposite ends of the data spectrum.

Quantitative Data

This is one end of the data spectrum. As a marketer, this is what you’re no doubt most familiar with — numerical or statistical data. Quantitative data like analytics: 

• Tells you what, where, when and how.
• Provides insights into performance and usage

It also provides a statistical foundation to track certain aspects of your website over time, such as:

• measuring clicks
• time on site
• tracking each user’s journey through the site
• where usage is improving/increasing, so we can amplify those features even more.

In certain cases, we can actually use analytical data to make assumptions about user behavior. For example, it shows us when users are spending a significant amount of time on a given webpage. It shows us exactly how much time users are spending on that page, but doesn’t tell us why — are they engaging more deeply with the page content, or are they frustrated because they can’t find the info they need?

To learn why, we’ll need to use data from the other side of the data spectrum — qualitative data.

Qualitative Data

Qualitative data helps us understand the ‘why’ behind user actions, and how users actually feel about your product. By incorporating qualitative data into your website analytics, you convert your users into advocates for your brand, organization, and mission. It also helps you discover ways to tell your story more effectively. 

Qualitative insights involve acquiring data from things like:

• Moderated or unmoderated testing of website components
• User surveys and interviews
• Examining interactions using real-time recordings.

Click-through rates (CTR) are a great example. If we notice that a button is frequently clicked, we might consider it a positive thing. However, this is where qualitative analysis comes into play. We need to dig deeper to understand the user’s intent behind their actions. It could be that users are clicking away because they’re frustrated. They can’t find the information they’re actually looking for — so they’re just clicking at the most visually prominent item on the page, hoping it will somehow take them to the info they seek. (This is sometimes referred to as ‘rage clicking’.) 

Qualitative insights allow you to put user needs first,  and align them better with your organization’s goals. They also provide a deeper understanding of how your product solves a problem for your users — and just as importantly, where your product might be putting up barriers that prevent users from solving problems for themselves. 

So, which is best for your organization? 

You guessed it. The answer is, both. 

Again, we’re talking about a spectrum, and the goal is to try and bring its opposite ends together. At Kanopi, we suggest our clients allow room in their budgets for a blend of qualitative and quantitative research. By looking at both types of data together, we can align analytical, business goals with user needs to create some really powerful experiences.

Types of qualitative testing

From a high-level testing perspective, we split qualitative testing into two main categories: 

1. Discovery-based testing, including interviews, focus groups, etc. Primarily done through observation, its goal is to understand the user’s reaction to a product, service or feature. This is where we gain a more emotional understanding about how users feel about the experience as they navigate your website.
   
   Discovery based testing allows us to better understand user motivations so we can create content that resonates and converts more effectively. This means using powerful techniques like storytelling, built on a foundation of research that includes focus groups, interviews and/or card sort testing.
   
2. Evaluative testing, which tends to be more task-based. We assign the test subject a particular goal, and then we ask them to accomplish that goal. We then observe how they accomplish it and record their feedback. This type of testing allows us to surface practical friction points and inefficiencies. During our observations, if we see users experience a moment of hesitation, we can immediately examine possible reasons why. Is the language not clear enough? Does the information architecture need to be revised?
   
   So while discovery-based testing helps provide content and design solutions that are more emotionally-driven, evaluative testing leads to more intuitive interfaces and more performative types of content — solutions that help users  move through your site more effectively.

How qualitative data can enhance quantitative data

Let’s look at a couple different ways to combine both types of data together to create a holistic picture of your users and your website’s functionality. 

Combining interviews with analytics

User interviews are a powerful way to understand how users are interacting with your product — and also how they feel about your product. A great example of this is bounce rate. 

Looking at bounce rate alone allows us to infer certain things. An abnormally high one suggests that something about a page or feature isn’t working for users. However, user interviews can provide clarity and context you’d never get from quantitative bounce rates alone. Sometimes, they reveal a solution that’s much simpler than you might expect — moving contact info up to the top of the page, for example.

Interviews help us map user goals and spot the barriers to those goals. But they also allow us to create targeted, phased improvements. We’re not trying to redesign the entire page or entire section of the site; we’re only looking to find the thing(s) that are actually needed to improve performance. You can make incremental changes over time, and test the results without having to do complete redesigns. 

Another reason we highly recommend combining user interviews with analytics is to help us choose where to place content and optimize key points along the user journey. They help us define content mapping, so we can prioritize certain content. They also help us spot gaps in our content — things that users are looking for but can’t easily find, or may not even exist. Similarly, they can help us identify content that should be archived.

Combining usability testing and traversal (treejack) testing

This is one of my personal favorite combinations of quantitative and qualitative testing. Traversal testing (frequently referred to as treejack testing) lets us observe how users navigate their way through your site.  It shows us which areas are ripe for optimizing conversions, which areas are being abandoned, and more.

Combining usability testing and treejack testing is a great way to determine if your website’s navigation is delivering optimal results. Over time, the art/science of website navigation has grown increasingly complex — and it’s become more challenging than ever to prioritize items in your navigation menu. Combined usability/treejack testing can provide valuable insights to help us structure our menus more effectively.

It also helps us create a more perfect balance between search engine optimization and usability, by including enough information that search engines are happy without adding so much that we overwhelm users with too many options.

Treejack testing consists of four main steps:

1. Defining our navigation structure
2. Establishing a set of user goals
3. Finding and enlisting real-world users to test
4. Observing their journey as they try and achieve those goals, making note of:
   • Points of frustration/confusion
   • Where things go smoothly
   • Opportunities to optimize their journey.

It also helps us better define horizontal pathways to content within web pages themselves, thereby eliminating the need to cram everything plus the kitchen sink into your navigation menu.

Lastly, usability/treejack testing helps us refine language, which is especially important when it comes to navigation. It gives us the opportunity to add clarity and focus, making the nav process more intuitive and helping us achieve our ultimate goal: Don’t Make Users Think.

Session recordings and heat maps

Heat maps are colored overlays that show high interaction points and user scroll behavior  helping us determine how effective our calls-to-action and content are. 

Using heat/click maps combined with session recording allows us to analyze data not just holistically but also at a granular level, and then modify our designs to provide a more seamless, intuitive experience. 

User surveys and A/B testing

As the name implies, A/B testing involves presenting users with a choice of two options and having them choose their preference. It can be used to test  content, functionality, visual design/user experience, and more.

At Kanopi, we mostly use A/B testing as a comparative test to validate a design direction; then we layer in additional context by conducting user surveys to gain some insight into the reasons behind their choices. 

We do this by separating users into two cohorts: we’ll do A/B testing with one, and user surveys with the other. We prefer this method since A/B is a comparative test between two options, and users are simply choosing what they feel is the better of the two. However, the option they’ve chosen still might not be the perfect solution — hence the need for a second cohort. User surveys offer a fresh set of eyes, to identify unforeseen problems that a straight-up A/B choice won’t tell you.

By combining the quantitative and qualitative testing, we get two levels of refinement. 

Qualitative user testing on a budget

Clients often have a perception that qualitative testing is expensive. So, let’s dive into this:

 How can you perform qualitative testing on a budget? 

In some ways, the notion that qualitative testing is expensive is true; analyzing data can be time-consuming. Finding users to participate in testing is one of the areas where clients express a lot of trepidation, so let’s list some of the ways that you can leverage the resources that you already have within your organization: 

Leverage your email lists and/or direct relationships. Start with your marketing email list. And/or, if you’ve got boosters such as donors or volunteers, reach out to them (and their network of people). 

Incentives are also a great way to encourage participation. It’s funny how a simple $5 Starbucks gift card can motivate someone to complete a five-minute test. (This is another thing to keep in mind: qualitative testing doesn’t need to be a lengthy undertaking. It can often take only 5-10 minutes.)

Don’t forget your internal stakeholders! The great thing about leveraging folks for internal testing is that they most likely already interact with your website fairly regularly, so they’re aware of  its ups and downs. The not-so-great thing is that there may be some internal bias you’ll need to mitigate. The way we handle this is by phrasing our language with the user in mind. For example, instead of saying “what color do you prefer?”, we’ll ask “which color do you think will resonate more with users?” This helps shift their focus from an internal bias to a more user-centric mindset.

The magic number

Another thing we find is that marketers tend to be more familiar with analytical data sets, where getting the best data requires a large pool to draw from. However, with qualitative data you don’t need a huge amount of user testing. Here’s the magic number to keep in mind:

Seven.

If you can get seven people to test a feature, product, or functionality on your site, they’ll provide you with all the actionable insights you’ll need.

Doesn’t seem like seven would be enough people, does it? But here’s the thing: we’ve done qualitative tests using literally hundreds of people. And we’ve learned that once you get over seven testers, all you’re doing is hearing basically the same seven responses over and over. So, if you’re concerned about your budget, or finding enough users, or any other scope-related issues, just keep that magic number seven in mind. It’s all you need to gain meaningful qualitative insights. 

DIY data analysis with AI

AI tools for data analysis are starting to come online. At Kanopi, we’ve started using a combination of some of these tools.

ChatGPT helps us develop prompts that we can then plug into other AI platforms like harpa.ai. We’ve found that ChatGPT is a great way to develop language we can use to speak to other programs. (And yes, at the same time we also find the prospect of robots talking to robots to be a tad unsettling. But, good data is good data.) Anyway, we find that combining AI tools in this manner really cuts down on analysis time as well. 

Also it should be said that online platforms are not only incredibly price competitive, they also provide you with a suite of tools. In some cases, for things like user testing, these platforms will actually go out and recruit users to test your site for you. So if you’re trying to work within a tight budget, look into online platforms like UserTesting, UserZoom, Userlytics and Trymata. 

Go forth and be iterative

You can apply the insights gained from qualitative  research using a phased or iterative approach — you can try it on something small and then iterate upon that if it’s successful. 

When DIY is not enough

I’ll leave you with some handy resources you can use for your qualitative research journey. 

Of course, we realize that for any number of reasons, DIY is not for everyone. And there are definite benefits to working with a team of experienced professionals who can seamlessly turn both qualitative and quantitative research insights into engaging content and design, that will help forge a stronger connection between your organization and your donors, supporters, and users in general. And if we were networking face-to-face at a conference, this would be the point where we’d politely present you with our business card, and mention that in addition to strategy and design we also provide full-service development and ongoing support. 

Support Tools

Survey Tools

• Google Forms (Free)
• Survey Monkey (Free up to 10 questions) 
• SurveyGizmo (Free up to 3 surveys)
• Typeform (Free 3 surveys/month, up to 10 questions)
• Survicate (Free up to 100 responses/month)
• Survey Legend (Free up to 3 surveys)
• YesInsights (7 day Trial)
• Google Surveys (Price range from 10 cents to $1 per response)

Usability Testing Tools

• UsabilityHub (Free up to 2min/test)
• Qualtrics (Free up to 10 questions)
• UXtweak (Free Starter plan)
• Lookback (14 day trial)
• UserTesting (Free trial)
• Userbrain (Free trial)
• Loop11 (14 day trial)
• FocusVision (Free trial)

Session Recording Tools

• Hotjar (Free with limited data storage)
• Mouseflow (Free 1 website, 500 recordings/month)
• CrazyEgg (30 day Trial)

Card Sort Tool

• Optimal Workshop (Free up to 30 cards sort, 10 responses p/study)