Analytics Blog

Consider Realism When Designing Choice Experiments, Especially With Small-Screen Devices

Replicating actual market decisions in a survey choice task is an important design goal. However, given that survey respondents have moved to devices with smaller screens, it has become increasingly difficult to accomplish the goal of realism.

For this reason, a summary of the most important elements of realism becomes critical, so that the choice task design can focus on incorporating only these most important elements. Where space is limited, the design must be focused on what is most important to producing realistic choice model outputs.

What are the most important elements of realism that we should consider in the design of the survey choice tasks? The list of important elements is certainly debatable, and experienced choice task designers will have different opinions and experiences of success.

Here is my list:

1. Including the brands and products most likely to be purchased
2. Including a range of products and prices within the product line of each brand
3. Building in context effects where feasible

Including Brands and Products Most Likely to Be Purchased – Most purchase decisions are made with a consideration of alternative brands and products. It is very difficult to know what brands and products each customer considers prior to making a final purchase decision.

For retail products, the variety of shelf arrangement across stores, the proliferation of online commerce and the variety shopping journeys (across stores and online) implies hundreds or even thousands of possible competitive sets. For products and services that are purchased based on conversation with a sales representative, there is generally a great variety in practice in selecting which options to offer a prospect. Many product categories could be given to drive this point home.

Given the difficulty, if not impossibility, of customizing a competitive set to each individual respondent, a best practice is to always include those brands that account for at least 80 percent of market share, since these brands have a higher probability of being considered by most customers.

With smaller screen sizes, typically, only two brands can fit column-wise within a single choice screen. However, all brands can be shown across all screens for the same respondent. While not every brand is currently considered by every customer, and some customers may consider only one brand, every brand has a non-zero probability of being considered by each customer. Brand preferences almost always have an unobserved component that can be explained, at least partially, by chance. For this reason, I recommend exposing each respondent to all brands across choice screens.

Some customization of brands shown can be helpful. If brand loyalty is strong in the particular category, one may program survey choice tasks to always display the respondent’s current brand on every choice screen. Alternatively, the current brand could be shown on more screens than would other brands.

Including a Range of Products and Prices Offered within the Product Line of Each Brand – Whenever a brand is displayed, a range of its product line should appear. For example, a small, medium, and large bottle of ketchup, with appropriate pricing, should appear in a choice screen, if all three sizes usually appear on a retail shelf.

One should avoid dominance within each brand, because in the actual market, dominance is generally avoided by sellers. For example, two products of the same brand, one being superior in functionality and lower in price, would not typically appear on the same shelf.

Of course promotions (e.g., Buy 1 and Get 1 Free) do cause temporary dominance, but this could be handled by adding a promotion attribute to the choice design and varying promotion level independently with price.

Building in Context Effects Where Feasible – In purchase occasions for some product categories, an attractive trade-off between two attributes is used to encourage customers to upgrade to the more expensive product.

Suppose, for example, you are buying new tires and your online research suggests the following market summary or benchmark information:

• 40,000-mile tread life for $50
• 50,000-mile tread life for $60

Next, you shop for new tires and are given two options:

• 40,000-mile tread life for $55
• 65,000-mile tread life for $65

In this case, the second option (65,000-mile tread life for $65) looks very attractive vs. the benchmark information, since this option provides another 25,000 additional miles for only $10 more – much better than the benchmark of 10,000 additional miles for $10 more. In order for the choice study to realistically mimic this type of market reality, a constraint on the experimental design (or randomized design) could be imposed to produce options with an average of 1000 miles per dollar across all scenarios, and an interaction term between tread life and price could be estimated in the choice model to capture the upsell utility for options that provide an attractive trade-off of miles vs. price.

Compromise is another behavior that consumers practice. In actual buying scenarios, consumers often gravitate to a mid-tier offering. This behavior occurs when at least three tiers are offered within a product line and is frequently utilized by sellers to maximize their brand’s market share.

Context effects should be considered whether choice screens are larger (desktop computer) or smaller (smartphone or tablet). I would argue that context effects are even more important in the case of smaller screens, because the design and modeling must be very focused on just the most important elements of realism.

Conclusion

Designing choice screens for smaller-screened devices is challenging, and simulating realism is difficult. Yet, realism is a key advantage of choice modeling, so developing choice task screens for smaller devices should focus on retaining the most important elements.

One way to include the most important brands, exposing all of them to each respondent, is to use only two brands per screen, but to ensure that all of the brands appear across all screens. On larger screens, the practice has been to expose each respondent to eight to 15 screens. With smaller devices, this rule of thumb still holds. The number of screens should not be reduced since eight to 15 screens are required to measure individual-level utilities, a measurement made possible by hierarchical Bayes or latent-class model estimation methods.

Using as many as 15 screens in the case of small-screen devices could be challenging, as survey respondents may struggle with the repetitive and time-consuming evaluations of multiple scenarios. One way to use as many as 15 screens in a survey conducted over a small screen could be to spread the survey across multiple instances, separated by time. For example, the respondent could evaluate five screens per instance, with three instances which would be completed over one week.

This approach may actually produce more reliable results than exposure to 15 screens in one instance, since evidence has been found that respondents begin to use artificial simplification strategies in order to get through 15 screens faster, such as focusing more on price and less on other features.

Including a range of products per brand might be best accomplished by displaying the brands (two of them) as columns, but displaying the products, offered by each particular brand, vertically down the small-device screen (i.e., as rows). A product line of three products per brand can be displayed with downward scrolling, which is easier than scrolling to the right.

The ability to show at least three products per brand is a must, since many of the context effects, such as the compromise effect of moving to a mid-tier offering, is a common consumer behavior. In order to capture this type of consumer behavior, choice tasks must display at least three products per brand.

The advantages of choice modeling in virtually simulating actual buying decisions with strong elements of realism are too great to abandon simply because survey respondents have moved to tablets and smartphones. More work should be done to optimize choice modeling surveys for smaller-screen devices.

Many of you may have conducted valuable research already in some of the areas mentioned. I welcome your comments and discussion.