Questionnaire Blog

Going Off The Grid: Not Just For Hippies & Outlaws

What do you think are the top frustrations for survey-takers? Endless open-ended questions? Poorly worded questions or confusing survey flow? Surveys being too long?

As a marketing research professional, and a customer who takes surveys, I have seen each of these issues crop up. But today I’m focusing on the over-used and abused question format—the grid question.

You know the one. In the first draft of your questionnaire, you create a few questions with rating scales. Each one has five or six attributes. By the time you review the qualitative results again and find one more nugget, and then your boss and the CMO add their input, the final questionnaire has monster grids that are as long as the ingredient list of your favorite processed cheese product.

I’m not suggesting a moratorium on using grid questions. With my background in customer satisfaction, I have a fond place in my heart for them. Often they are unavoidable. And sometimes they can be the most efficient way to get at a comparison of a small set of attributes. But we all know that the “just one more” attitude can quickly produce a glut of data that we aren’t likely to use to its fullest potential. And what’s worse—data quality and survey length are inversely correlated, so those ever-growing grid question may actually be weakening the quality of the input you ultimately receive.

How do you know it’s too much? One measure I use to determine if I should consider paring down the list is my pain level during the review phase. During an early review of the programmed survey, I gauge my frustration level. If I get frustrated going through a grid question on the second or third pass, when I’m not even focusing on the content, I pause. How do I expect real respondents to give thoughtful attention and weigh each attribute carefully if I’m annoyed just clicking through to get to the next screen? Real talk: they probably won’t.

Paring down the grid is one way to improve the survey experience. Is there another way?

If you know that you need to compare, rate, or optimize a long list of features or attributes, consider using MaxDiff (Maximum Difference Scaling) instead of a grid. MaxDiff forces respondents to make trade-offs (or choices), like they do in the real world. MaxDiff is more engaging for respondents and simulates trade-offs in the buying process. Since a long list of attributes is presented a few at time over multiple screens, it’s also much less taxing than a wall of attributes on a grid. There are advantages for you too. Instead of finding out that everyone thinks that having 15 grams of protein, being dairy-free, and being made with organic ingredients are “extremely important” when deciding which morning beverage to purchase, why not get more out of your survey data? Wouldn’t it be better to know that protein is twice as important as organic ingredients?

MaxDiff may require a larger base size or an additional cost for the project, but it is well worth the investment in terms of the quality of results and useful insights you get in return.

Just remember: Long grids? As if. Choose MaxDiff!

If you’d like to learn more about this approach (and other choice modeling options), watch the Choice Modeling Video (at the top of this page) made by people who have more to offer than a funny slogan.

Jennifer Murphy

Vice President

Jennifer has over 20 years of experience in marketing research in a wide variety of roles, from data processing to reporting and analysis. Her attention to detail and big picture thinking, combined with her quick wit and deep knowledge of research, makes her a wonderful project collaborator. Jennifer earned a Bachelor of Business Administration in Marketing from the University of North Texas and a Master’s in Marketing Research from the University of Texas at Arlington.

• 
•