When do you use quantitative vs. qualitative research?

6 Replies

As product managers we use research to inform decisions and direction of the product. And there are many types of research you can do: user research, competitor research, and market, research. 

But really most product research can be divided into two main categories: quantitive and qualitative.

  • Qualitative research = open-ended questions > hypothesis 
  • Quantitative research = data-centric > validation

So, before you immediately start wrangling data or setting up user interviews ask yourself one simple question:

Would my problem best be solved by feelings or facts?

Once you understand your goal, you can leverage the right type of research.

Qualitative: Data is subjective and typically collected through user interviews or focus groups. Qualitative research requires you to pull out themes from comments or descriptions.

While generally harder to draw conclusions and interpret than quantitative research, qualitative research can lead to a deeper understanding of motivations. Hearing the way users or customers describe their problem in their own words can result in insights that data alone masks. 

Most useful for identifying problems and developing a hypothesis.

Quantitative: Data is objective and typically collected through A/B testing, analytics tracking tools, or even surveys.

Quantitative research is usually easy to interpret and can help quickly confirm or refute assumptions. But only if you already know the question you are trying to answer. 

Most useful for validation.

Good product managers will depend on both quantitative and qualitative data to get the full picture. But understanding the right time to use each is a powerful skill every product manager should strive to master. 

Quantitative best when you need to determine to true value of something: "How much is it worth to you?" "Now how much would you pay? 2X? 3X? " :-)

They're complementary. One tends to be useless without the other. Having information that says "X occurrences happen" is typically useless without understanding what happened in those occurrences, how the user felt, what they felt they couldn't accomplish, and what they were surprised at what they accomplished.

Quantitative is "easier" because it's something that has hard data with 1's and 2's. Qualitative is hard to get people to want to use because senior management typically position themselves to be unable (really it's unwilling) to turn the questions from customers or other executives around into a "I can give you data but it's not painting the whole picture" conversation. Often times folks look at qualitative data as anecdotal, which it certainly can be if the research is done poorly (i.e. using poor methodology).

It's confusing phraseology because qualitative research still quantifies and quantitative research may still qualify. For example, qualitative research may tell you that people feel your message is depressing. But do I care if only 1 person in a focus group feels that way and everyone else finds it uplifting? No, I weight the qualitative by quantifying it. But I may conduct a survey that asks a question like "How well did this feature meet your needs?" with multiple choices like "It didn't , it was overly simplistic." This only works if I feel very confident that the qualifying elements are the only elements. In this example, that overriding qualitative issue was the feature's level of complexity.

So then, to answer the question, I use Qualitative Research to find out the broad views my subjects bring to the issue I am trying to understand. I want to approach the learning from many different directions and in many different ways to maximize my learning. I can then use Quantitative Research to quantify the Qualitative feedback my measuring how many times a qualitative element is or is not perceived, and the weighted value of each of these perceptions vs other perceptions. If you feel very confident as to the various perceptions effecting your issue, you may jump straight to Qualitative.

Of course, some item are just pure quant "How many elearning courses did you take in the last 12 months?" For these a survey can be executed right off. It may make sense to use a quantitative first to identify a specific type of subject and followup with a qualitative focus group to get that cohorts view.

Hope this helps!

When ideating on the MVP of a novel idea/solution, I tend to rely on qualitative research 'coz most times there is no prevailing evidence to whether it would succeed or not (only proxies and gut feeling). But post MVP, I rely on quantitative research to validate the uptake/ response/feedback from actual users before I invest any more time or resources.

However if the MVP itself require a huge commitment (in terms of effort or resources), quantitative data is a huge help in determining the relevance and quantum of the business case.

Qualitative methods help you identify what you don't already know; quantitative helps you measure what you know. In addition, some methods are best for discovery while others are best for validation.

Before you begin, you should search for insights from existing research. You can often infer insights from secondary research even when it is not done with your specific goal in mind.

This chart from one of my workshops shows my preferred methods and where they are used. Every product manager and product owner should interview and observe customers. There simply is no substitute for first-hand customer experience. And no excuse for not doing it. Once you have a hypothesis, you can use the other methods to provide additional insights and validate potential solutions.

For more on interviewing, see "Customer Interviews: A Field Guide" at http://under10playbook.com/ebooks/customer-interviews