Sampling Blog

Who’s taking my survey?

I recently attended The Quirk’s Event in Brooklyn. As always, the conference was well organized with a variety of topics being covered and valuable learning being shared, but one particular session that I attended had me raising my eyebrows and saying “Wow.”

The focus of the session was panel and sample quality, and the challenges facing our industry today (pre COVID-19, of course). I was not surprised by the challenges we’re facing, but was stunned by the poor quality of the data this speaker encountered.

Panel and sample quality is something all market researchers should be concerned about and actively engaged in promoting, but based on the information shared during the session I attended, it may not be getting as much attention from all researchers as I had assumed. So, with that in mind, I thought perhaps it was time to take a step back and consider what we are doing to make sure we can answer the question, “Who’s taking my survey?”

At Decision Analyst, panel and sample quality assurance takes place at a number of key points during the research process, starting with panel recruitment and continuing with sample selection, data collection, and data cleaning. Panel recruitment is a major undertaking, but it’s not my focus for today. Instead, today’s thoughts focus on sample selection, data collection, and data cleaning; and what we can all do to make sure our data is clean and provides valid results.

• Identify duplicate respondents. It isn’t a secret that the relationship between panel company and panelist is not exclusive. An active member of one panel is also likely to be a member of at least one or more panels. With many studies blending panels for a variety of reasons, the likelihood of a respondent being invited to the same survey more than once is a very real possibility; but again, this is something that we can guard against. Ideally, in the initial sample pull, checks are being conducted to reduce duplicates. In addition, during data collection, digital fingerprinting can be used to capture and clean out duplicate responses as well.
• Survey real people. Of course, this seems like a no-brainer, but bots and click farms are a real threat, and not just for placing “likes” on a Facebook page, but also for posing as real respondents to online surveys. There are several techniques to combat against this threat, including various technology options. Yet our most effective method is including survey specific open-response questions that the respondents must answer with details that reflect the survey topic. These questions cannot easily be answered by a machine. After, or preferably, during data collection, responses to these questions are reviewed by real people (who are more accurate and thorough than text analytics or a machine); and fraudulent respondents can quickly be identified and removed from the survey data.

While the hope is that the two checks above will capture any fraudulent or duplicate responses, we always employ additional steps to further improve sample quality.

• Set attention checks. If you’ve taken care of business in the sample pull, your respondent pool should include mostly valid respondents, but even good respondents can have a bad day. Adding one or two simple questions unrelated to the survey topic, and asking the respondent to perform a specific task, is a great way to make sure the respondent is actively engaged in the survey and providing thoughtful responses. If they aren’t, remove them from the sample.
• Remove questionable respondents. In addition to catching inattentive respondents, also look for those who are sloppy, speeding, or straightlining through the survey. In some instances, this may require manual checking of the data; but taking the time to do so results in better quality data and, therefore, better decisions from the research results.

Sampling quality assurance, driven by fraud-prevention efforts, is not an easy task. It can’t all be automated, which means someone has to make time in their day to perform these tasks. If you’re the end client having to perform these checks yourself on a full data set, it can be an overwhelming task and frankly, it shouldn’t be your job. I’m fortunate that Decision Analyst holds data quality in the utmost importance and employs a large staff of quality-assurance team members whose only job is to make sure we provide our clients with clean, accurate data from real people.

So, who’s taking your surveys?

Julie Trujillo

Senior Vice President

As a research consultant with over 20 years of experience, Julie provides strategic direction to clients who are struggling with difficult business challenges. Julie guides her clients to identify the business questions and then designs research to get answers. Julie’s experience includes quantitative and qualitative research across an array of categories and research techniques. Julie holds a Bachelor of Science in Business Administration, with a concentration in Marketing, from Valparaiso University.

• 
•