Direct Comparison Product Testing

Many inhabitants of the marketing world have heard of the term “paired comparison” product testing, and some may have heard of the term “triangle testing,” or “triangle taste testing,” and some are familiar with the term “Product Clinics.” All three of these methods involve direct comparison of one product to other products.

Paired Comparison Testing—target market consumers are given two products to taste and then asked to choose the one they prefer, or to choose the one that tastes better.
Triangle Test—target market consumers (or tasting experts) are given three products to taste (two are the same, one is different) and are asked to choose the one that is different.
Product Clinic—target market consumers are recruited to a central location to compare a test product to several competitive products. Typically, these are durable goods.

We will discuss each of these direct comparison testing methods in turn.

Paired Comparison Product Testing

Food is the assumed product category for paired comparison product tests and triangle taste tests, but almost any type of consumer product can be tested by some type of head-to-head, direct comparison. Paired comparison product testing is a widely used product testing methodology. Most often, target market consumers are invited to a central location (an R&D lab, a restaurant, a research facility, a test kitchen, etc.) where the two food products can be prepared and tasted by target market consumers. Alternatively, the two food products could be shipped to consumers’ homes for an in-home paired comparison test, although there is always a risk some consumers will confuse the two products when reporting the results.

The following procedures are put in place to level the playing field, so that each product has an equal chance of winning. The two food products should be equal and identical in every way, except for the variable or variables being tested.

Generally, the two products are tested “blind”; that is, all brand identification is removed or hidden.
The code names or code numbers for each test product are also hidden, if possible, so the consumer is not biased by the product code itself. For example, if the two products were labeled as “A” and “B,” the code letters might suggest to consumers that product A is better than product B.
The order of seeing and tasting the products is rotated from respondent to respondent, to eliminate any potential order bias, and the products are rotated from left to right and right to left, from respondent to respondent, to eliminate any positional bias.
The two products are served at the same temperature and in identical containers or on identical plates.
Respondents might be asked to drink a little sip of water and take a small bite of an unsalted cracker between tastings of the two products to “cleanse” the palate.
Respondents then taste each product one time or multiple times until a “preferred” choice is reached.
The prices of the two food products are almost never revealed before the paired comparison test. Prices could bias quality perceptions and skew the results.

In crafting the questionnaire, if you have two product formulations and must choose only one to manufacture, you would likely force the respondent to choose the “preferred” formulation. However, if the goal is to determine how similar two products are, then you might add a third answer choice (“no difference” or “taste the same”). In the latter instance, you may want to know if the two formulations are close enough to each other that you could substitute one for the other, depending on the price of ingredients.

Interpreting the results from paired comparison product tests can be tricky. For example, if you tested spicy ketchup versus a bland ketchup and 55% of the participants preferred the spicy ketchup while the other 45% preferred the bland ketchup, would you conclude that one of the ketchups was better than the other, or would you conclude that 55% of ketchup consumers prefer spicy ketchup and 45% prefer bland ketchup? We have to think carefully about exactly what we want to do with the results, and about how we will interpret the results.

The Triangle Test

This testing method is most often used in the food, health and beauty, and personal care product categories. The triangle test is most often used to determine if a substitute ingredient results in a product that tastes different from the current product. For example, let’s suppose that you manufacture a picante sauce, and the price of jalapeno peppers is rising rapidly, so you want to reduce the number of jalapeno peppers in each jar—but only if your consumers can’t tell the difference. This is a perfect case for the triangle test. And, you don’t have to use consumers off the street—they are very expensive; you can use your employees, and just to be safe, only use those who have proven that their tastes are sensitive to very small differences in ingredients and flavors.

By using sensitive expert tasters, you have stacked the deck against making a change in any ingredient (as you should—it’s a safety measure). Now you are ready to experiment. Assemble the taste-sensitive panel of employees and give each of them three containers of picante sauce (two are identical, and one is different). The latter contains fewer jalapeno peppers. Follow the same rules used for paired comparison tests; that is, eliminate all possible sources of bias and strive to make each of the three products the same in all respects, except for the variable being tested (in this case, the level of jalapeno peppers).

Rotate the order of tasting (and left-to-right positions), and serve all three products at the same temperature, etc. Ask the taste panelists to identify the one product that is different from the other two. They could choose product one, product two, product three, or a fourth choice of “can’t tell any difference; they are all the same.” Of course, participants would not see the product codes “one, two, three,” because of potential bias from the code numbers. To see if the group of employees can actually perceive a difference between the three products, we first subtract out the “can’t tell any difference” employees and then compare the number of “it’s different” choices among the three products.

If the panelists can’t really tell any difference, then the choices for product one, product two, and product three should be roughly equal (or within the margin of error allowed by statistical significance testing). The greater the number of panelists who say, “no difference,” the greater the likelihood that the average consumer would not be able to tell the difference if the formulation with fewer jalapeno peppers was substituted for the current formulation.

The triangle test gives you no information about whether the different product is better or worse than the other two products (the “controls”), but it could give you the confidence to substitute a cheaper ingredient or ingredient level—if consumers or expert panelists could not detect a difference in jalapeno levels.

Product Clinics

Product clinics are another type of head-to-head comparison among competitive products. These clinics are commonly used to compare durable goods (cars, washing machines, lawn mowers, tractors, etc.) to their major competitors’ products. Product clinics can also be used to compare consumer package goods with their respective competitors—but product clinics are rarely used for these product categories.

Let’s say your product category happens to be refrigerators. Here are some steps that could be involved in a refrigerator clinic.

You would set up a “show room” with 5 to 10 new refrigerators (your new refrigerator plus the new refrigerators made by major competitors).
Next, invite target market consumers (e.g., consumers who planned to buy a new refrigerator in the next 12 months) to the show room to evaluate the new refrigerators.
As with paired comparison and triangle tests, it’s important to implement the same types of controls and take precautions to eliminate potential biases, so that the comparisons are a fair and balanced evaluation of the various refrigerators.
Most clinics tend to be diagnostic in character, so it’s typical to ask consumers what they like about each of the refrigerators and what they do not like about each.
After the consumers evaluate the refrigerators one by one, they are asked which one, if any, they would prefer to buy. They might be asked for a second choice, or even a third choice. Of course, respondents would be asked to explain why they chose each refrigerator.
Lastly, you might introduce prices, to see how preferences would change as a function of price.

Paired comparison, triangle tests, and product clinics are the primary direct comparison product tests, and there are many variations on these basic research designs. Product testing is one of the most valuable types of marketing research, and it almost always yields actionable information to help a product or brand gain competitive advantage in the marketplace. Direct comparison product testing should be a part of your research toolkit.

Author

Jerry W. Thomas

Chief Executive Officer

Jerry founded Decision Analyst in September 1978. The firm has grown over the years and is now one of the largest privately held, employee-owned research agencies in North America. The firm prides itself on mastery of advanced analytics, predictive modeling, and choice modeling to optimize marketing decisions, and is deeply involved in the development of leading-edge analytic software. Jerry plays a key role in the development of Decision Analyst’s proprietary research services and related mathematical models.

Jerry graduated from the University of Texas at Arlington, earned his MBA at the University of Texas at Austin, and studied graduate economics at SMU.

Product Testing Blog