Product Testing White Paper

A Systems Approach. The methods and procedures of product testing should constitute a standardized system, so that every like product is tested exactly the same way, including:

• Identical product preparation, age, packaging and coding.
• Identical questionnaires (of course, parts of the questionnaire must be adapted to different product categories).
• Identical sampling plans, typically employing blocking-screening grids to ensure matched samples.
• Identical data preparation and tabulation methods. Similar analytical methods

Normative Data. As products are tested over time, the goal is to build normative databases, so that successive product tests become more meaningful and valuable. The normative data, or norms, continually improve a company’s ability to correctly interpret product-testing scores, and the norms help reveal exactly how good, or how bad, the test product is.

Same Research Company. Use one research company for all of your product testing. This is the only way you can make sure all tests are conducted in exactly the same way.

Real Environment Test. If the product is used in offices, it should be tested in offices by people who work in offices. If the product is typically used at home, it should be tested at home. If the product is consumed in restaurants, it should be tested in restaurants, and so on. In general, this kind of “real environment” test will produce the most accurate results. For example, for food products an in-home usage test is almost always more accurate and predictive than a central-location taste test.

Relevant Universe. Sampling is a critical variable in product testing. For new products or low-share products, the sample should reflect, or represent, the brand-share makeup of the market. For well established, high-share (or highly differentiated) products, the sample should contain a readable subsample of that product’s users and a readable cell of nonusers. If the product category is underdeveloped (e.g., a relatively new category), then the sample should include nonusers of the category, as well as users. Also, it’s always important to represent medium to heavy users of the product category in the final sample.

In summary, if a company’s brand share is very low, it’s important to assign more weight (or importance) to the opinions of nonusers of the brand. If brand share is very high, then what brand users think is more important.

Critical Variables. Product performance and quality must be defined from the consumer’s perspective, not the manufacturer’s. What aspects of the product are truly important to consumers? What critical variables determine the consumer’s satisfaction with the product? These critical variables must be identified for each product category (typically, with focus groups or depth interviews) to design an accurate product-testing system.

Conservative Actions. The formulation of an established product should never be changed without careful testing and evaluation of the new formulation. Once you are sure you have a better product, introduce it into a limited geographic area for a reasonable time period (several product repeat purchase cycles). Then, and only then, roll the new product out to all markets. The smaller the market share, the greater the risks that can be taken with a new formulation. The larger the market share, the more conservative one should be in introducing a new formulation.

Monadic Testing typically is the best method. Testing a product on its own offers many advantages. Interaction between products (which occurs in paired-comparison tests) is eliminated. The monadic test simulates real life (that’s the way we usually use products—one at a time). By focusing the respondent’s attention upon one product, the monadic test provides the most accurate and actionable diagnostic information. Additionally, the monadic design permits the use of normative data and the development of norms and action standards. Virtually all products can be tested monadically, whereas many cannot be accurately tested in paired-comparison designs. For example, a product with a very strong flavor (hot peppers, alcohol, etc.) may deaden or inhibit the taste buds so that the respondent cannot really taste the second product.

Sequential Monadic Designs are often used to reduce costs. In this design, each respondent evaluates two products (he or she uses one product and evaluates it, then uses the second product and evaluates it). The sequential monadic design works reasonably well in most instances, and offers some of the same advantages as pure monadic testing. One must be aware of what we call the “suppression effect” in sequential monadic testing, however. All the test scores will be lower in a sequential monadic design, compared to a pure monadic test. Therefore, the results from sequential monadic tests cannot be compared to results from monadic tests. Also, as in paired-comparison testing, an “interaction effect” is at work in sequential monadic designs. If one of the two products is exceptionally good, then the other product’s test scores are disproportionately lower, and vice versa.

Paired-Comparison Designs (in which the consumer is asked to use two products and determine which product is better) appeal to our common sense. It’s a wonderful design if presenting evidence to a jury, because of its “face value” or “face validity.” The paired comparison can be a very sensitive testing technique (i.e., it can measure very small differences) between two products. Also the paired-comparison test is often less expensive than other methods, because sample sizes can be smaller in some instances.

Paired-comparison testing, however, is limited in value for a serious, ongoing product-testing program. The paired-comparison test does not tell us when both products are bad. The paired-comparison test does not lend itself to the use of normative data. The paired-comparison test is heavily influenced by the “interaction effect” (i.e., any variations in the control product will create corresponding variance in the test product’s scores).

The Protomonadic Design (and the definition of this term varies from researcher to researcher) begins as a monadic test, followed by a paired comparison. Often sequential monadic tests are also followed by a paired-comparison test. The protomonadic design yields good diagnostic data, and the paired comparison at the end can be thought of as a safety net—as added insurance that the results are correct. The protomonadic design is typically used in centrallocation taste testing, not in-home testing (because of the complexity of execution in the home).