Sampling Blog

Ergo, the first deadly sampling sin is the “race to the bottom.” Corporations and research agencies which consistently select the lowest-priced online samples tend to drive the prices that online panels can charge lower and lower. This in turn reduces the money available to online panels for recruiting good participants; for rewarding panelists who complete surveys; and for implementing all of the quality-assurance procedures, software, and systems necessary to ensure high-quality online samples. This “race to the bottom” also causes companies to turn to lower-cost, dynamically sourced “river” samples and the attendant skews and biases that might be unknowingly introduced into surveys.

The second deadly sin is routing. Before acceptance into a survey, respondents may be “routed” through several separate screeners in attempts to qualify them for participation in various surveys—these screeners are sources of potential bias and, possibly, fatigue. Then, once participants complete one survey, they might immediately be routed to a second survey, or even a third survey. Routing dramatically reduces the cost per completed survey, but it injects unknown biases into survey results. The first survey might bias answers to the second survey and the third survey, not to mention the effects of respondent fatigue. Routing is widespread in the research industry, so it’s something to be aware of and discuss with your sampling partners. Clients’ demand for cheaper samples is one reason for the rise of routing; a second reason is a shortage of high-demand groups of respondents (young males, for example).

The third great sin is speed. If a survey is posted and massive amounts of sample are released, the survey is completed very quickly, sometimes within an hour or two. All of the quota buckets fill up fast. This can bias the results, because the folks who spend more time online are the ones most likely to respond quickly. These fast responders might not be representative of the whole population. Releasing too much sample too quickly can also introduce geographic bias. If all of a nationally representative sample of U.S. is released at 5 p.m. Eastern time, the various quota buckets will fill up with residents of the Eastern time zone, since they get home and start answering surveys before people in the other U.S. time zones. If at all possible, slow down the sample release and trickle the sample out over a period of several days, so that everyone in the sample has an equal chance to respond.

The fourth deadly sin is un-nested quotas. For example, the Client or the research agency might specify age quotas and male-female quotas separately. When the sample is released, young women might be the first responders. So, young women fill up all of the younger age quota groups. When the slow-moving males arrive for the survey, only the older ones will be admitted into the survey. So, the resulting sample consists of young females and old males. This is not a representative sample. A better way is to nest (or stack) the quotas, so that you have “buckets” for young males, middle-aged males, and old males—and the same three buckets for females. You then end up with a completed sample that looks like the U.S. population.

The fifth sin is setting completed-survey quotas that introduce bias into the final sample. For instance, let’s suppose your goal is a nationally representative sample of Honda owners. You could look up the latest population statistics and set quotas by geography, age, and gender to create a nationally representative sample. Then, you find Honda owners who fall into these quota buckets. The problem with this approach, of course, is that Honda owners don’t actually fit these population distributions. Honda owners might be concentrated on the West Coast or in major urban areas, so this “nationally representative” sampling approach actually introduces distortion or bias into the final sample of Honda owners. A better way is to screen a national probability sample of U.S. aged 16+ population for ownership of Hondas. In other words, balance the sample at the screening level, and then let Honda ownership determine if someone falls into the sample. That would result in a true, nationally representative sample of Honda owners.

The sixth deadly sin is biased screening questions. It is not uncommon for companies or agencies to screen for a sample of consumers who, for instance, “like the idea of high-protein cereals” to evaluate a new product concept for a high-protein cereal. Everyone is happy because the concept test results are so positive, but the new product fails in the marketplace. Another example: a major U.S. automotive manufacturer chooses to exclude anyone who owns a Japanese-made car from the sample to study acceptance of new automotive concepts. You can see the potential for biased results with these types of “favorability” screening. That is, these types of screening are designed to hype positive results (sometimes on purpose, sometimes unconsciously). A national probability sample of product category users is a safer and fairer way to structure samples.

The seventh and last deadly sin is an unstable sampling universe. Let’s say you are designing a continuous advertising tracking study for chocolate candy. You could specify a sample of U.S. population 14+ in age; that would be a stable (or relatively stable) sampling universe. You could also specify the sample as U.S. population aged 14+ who purchased chocolate candy in the past 12 months; that would also be a stable sampling universe. Or, you could specify the sample as U.S. population aged 14+ who purchased chocolate candy in the past two weeks. This is an example of an unstable sampling universe, because your sampling universe would expand dramatically around Valentine’s Day and Halloween since almost everyone buys chocolate candy during these time periods. Strangely, you might notice that your brand’s advertising awareness falls during the time periods around Valentine’s Day and Halloween—without realizing the falling ad awareness is created by the influx of so many infrequent chocolate candy buyers during these time periods.