Research Sample Size: Confidence in the Data through Framing Questions

Ways of Framing Questions

Sample size isn’t the only thing that influences confidence. The way questions are asked in the survey can have an impact too.

If there are a few critical questions you need the research to answer for your marketing efforts, then you need to make sure the design of your custom research gives a very clear answer to those questions.

Suppose your company wants to know if your natural products brand should be more edgy and honest or more accessible and friendly.

You might think that you’ll get more information by asking respondents about 20 different directions the brand could go in, including the two above, and then seeing which ones rise to the top — just throwing everything in the mix and letting the research do its magic.

This is the “while we’re at it” strategy, where you add any possible option to the research in hopes of enabling more information to come from a single study.

However, asking about a number of different options at once doesn’t always work out that way. When respondents’ choices are spread out among so many different options, it might be hard to tell the difference between the two options you’re actually considering at the end of the day.

Particularly when you take confidence intervals into account, many of the options that got thrown into the research might end up being indistinguishable from one another and not significantly different from a statistical perspective. Asking too many options can result in situations where only a few people pick each option and differences between options are not significant.

Additionally, it can be harder for respondents themselves to decide which options they like more than the others when they have so many choices! Even aside from the “official” confidence intervals around the data, we start to get less confident about data when it’s coming from a question that can be difficult or tiring for respondents to answer in the first place.

To give an extreme example, imagine having to pick your top three sayings from a list of 100 sayings. That’s a tiring task, and you might just give up and pick the first three that look all right. If this question were fielded to several hundred people, it may appear that you could have confidence because of the sample size; however, the results may simply be the sum of many distracted or uncertain responses.

Ease of answering questions has to do with not only the number of responses listed, but also with whether these responses are distinct from one another and easy to understand.

When scanning over the questions quickly, perhaps even on a mobile phone or after having taken other surveys, respondents should be able to tell at a glance what each option means and how it’s different from the other options.

In our research, we identify and distinguish the most important questions your brand is facing for its next steps and plan research to answer those questions with a high level of confidence — not just statistically but also from a common sense perspective.

Reflecting on Research Sample Size/Confidence

Hopefully this three-part series has shown that sampling is a little more complicated than just getting the same number of respondents each time, throwing all the possible options into the survey, or taking every small difference in the final results seriously.

You want to get good answers for your brand and create effective segmentation as we do with our RealPersonas™ to help guide your marketing. But you want do so with certainty, and that comes when you maximize the confidence about your most crucial questions while staying within budget.

In an ideal world, you’d get to sit down with every consumer in your target population and ask them a million questions over the course of weeks and get accurate answers on each question.

But since that’s impossible, making smart decisions about sample size, segmentation, the source of the respondents, and how the questions are asked is the way to go!