Marketing Research White Paper

Customer Experience Mythologies:

The Net Promoter® Score Decomposed

The study of customer satisfaction, or customer experience, or whatever the latest moniker is, does not occur in a vacuum; typically, it takes place in the context of the large corporation.

Large organizations have tendencies, or peculiarities, that often complicate the process of measuring, understanding, and using customer satisfaction or customer experience data. Let’s look at some of these corporate complications.

Corporate Fantasyland. The typical corporate headquarters for a Fortune 500 company is a fantasy world where senior executives muse about their customers and about marketing to the distant masses. So the first great complication, or barrier, to understanding customers is boardroom fantasy. Many senior executives live in la-la land, far removed from the hustle and bustle of the marketplace. It’s difficult to provide great customer service, and develop sound marketing strategies, if you live in fantasyland.

Overpaid Executives. These high-level company executives often live in fancy houses, in gated communities, in protected enclaves, far from the ‘madding crowd.’ The pay of senior executives in Corporate America has soared into the stratosphere over the past 30 years, creating a vast separation or divide between senior management and their customers. It’s increasingly difficult for these overpaid and pampered senior executives to understand and relate to their customers.

Popular Books. Many senior executives are vulnerable to popular management and marketing theories because they don’t really know what to do; they are ‘out of touch,’ so they are more likely to grab on to popular theories or managerial fads—and create chaos within their organizations. CEOs read the latest popular books on business, such as The Ultimate Question with its Net Promoter® Score (NPS®), and often rain confusion and disorder down on their organizations. Senior executives should not be reading books; they should be out talking to their customers.

Complexity. Corporate executives attended the best colleges and universities, where they were taught the value of complexity and complex thinking. One’s ability to sound complex and complicated is often the currency of advancement in large corporate cultures. Consultants and gurus get hired because they, too, are selling complexity. All of this complexity leads to diffusion and confusion. Complexity is a barrier to successful execution and, hence, a barrier to serving one’s customers.

Youth Focus. If we were to read the marketing plans of Corporate America, we might well think that the average person in the U.S. died at the age of 35, or certainly by 45. It seems that marketing and advertising are biased toward the very young. It’s as though the rest of the population did not exist. There are 5,000 people turning 65 every day now, and will be every day for the next 20 years. We are approaching the point where 25% of the adult population in the U.S. will be over the age of 65. This focus on the young overlooks many marketing and business opportunities, and often reduces customer satisfaction, because corporations are ignoring a large part of their customer base.

Re-Org. Many of our largest clients (all Fortune 500 companies) rarely accomplish anything because they are in a constant state of upheaval, with re-org after re-org. The fact that we have this new word in our vocabulary (re-org) is testimony to its pervasiveness in practice. It’s not uncommon for our largest clients to reorganize every 12 to 18 months. During these episodes of turmoil, everything grinds to a halt. Projects are delayed. Decisions are deferred. Opportunities are lost. Customers are neglected.

Copycat. If there is one thing business people believe, it’s that major competitors ‘know what they are doing.’ That’s why they want to copy the competitors’ new products as quickly as possible, or copy their advertising, or copy their managerial fads. Go to an auto show and you will be shocked at how similar most new cars look, as though made from the same general mold. In product category after product category, you see the pattern of copycat similarity. These companies are looking at each other, studying each other, and ignoring the needs, wants, hopes, and dreams of their customers.

Trend Watching. Tracking trends via social media, surveys, and media hype is often sold as the way to be on the leading edge, to be innovative. By the time a trend is clearly visible, you are already too late and will seldom be able to catch up. If you are doing your research homework and talking to your customers, you can create trends rather than chase trends. If you do your research homework and really understand your customers you will know what is possible before everyone else knows. Let your competitors chase the fads and trends your company creates.

Tribal Tyranny. The tyranny of group pressure and group-think limits corporate creativity and effectiveness. Every company has its own culture, its own mythologies, its own distortions of reality. To be a member of the corporation’s tribe, you must accept and believe—you must ‘drink the Kool-Aid.’ The need to belong, to be a member of the tribe, is so strong that it can blind us to original thinking, blind us to our customers’ needs and issues. Tribal tyranny reduces corporate possibilities.

Fashion. Often invisible, fashion pervades all aspects of human culture. Fashion changes are highly visible in some industries (like women’s clothing, for instance), but difficult to recognize in most industries, because the pace of fashion change is slow and/or we simply don’t recognize it as a fashion. Every industry and every human activity is shaped and influenced by fashion and fashion trends, and many of these fashion trends are outside of our awareness. Corporations are hotbeds of fashion—managerial fashions, marketing fashions, accounting fashions, advertising fashions, presentation fashions, analytic fashions, etc. (Remember the corporate fantasyland). It’s key to understand the role of fashion in your industry and your business if you hope to keep your customers happy.

These complications are partial barriers to improving customer service and improving the customers’ experience. Let’s zero in on the world of customer satisfaction or customer loyalty or customer experience, whichever term you prefer (these terms overlap in meaning and use, but all recognize the importance of understanding and relating positively to customers). The many ‘truisms’ of customer satisfaction and customer experience lore are mostly a set of mythologies to deceive the gullible and exploit the innocent. Let us explore these mythologies and then talk about best practices for customer satisfaction and customer experience research.

The Customer Satisfaction Mythologies

Rising customer satisfaction scores are positively correlated with increases in sales. Sometimes this is true, but just as often rising satisfaction scores are inversely correlated to actual sales. For example, as sales go down, stores are less crowded and store employees have more time to spend with customers, and often retail prices fall as sales go down. If sales are declining, the marginal customers (the least positive) are the ones most likely to be lost each month. One or all of these factors can result in rising satisfaction scores’ while sales are falling. Retailers, in particular, can ‘happily’ go out of business amid a rising tide of customer satisfaction scores.

It costs less to keep your current customers than to attract new customers. This is usually true but not universally true. Keeping existing customers can become very expensive at times. Some customers are too costly to keep. The key is to focus on attracting and keeping the strategically profitable customers. Every business must do its homework to determine if spending to keep existing customers is justified by the facts.

Companies should strive to continually ‘exceed customer expectations.’ Really? Does this mean that marketing should set low expectations so that a company’s products and services can exceed expectations? If you provide great service one time, doesn’t that raise the expectations bar—so that the ratings next time are likely to stay the same or go down? Maybe a better strategy is for marketing communications to set high expectations, and for the company to work hard to meet those high expectations consistently.

Companies should always strive to maximize customer satisfaction ratings. Companies can achieve the highest possible customer satisfaction scores by raising product quality, offering unlimited service, providing lifetime warranties, and lowering prices. But these actions will certainly lead to bankruptcy. No, the goal should not be to maximize customer satisfaction. The goal should be to optimize customer satisfaction so that the company’s long-term profits are maximized.

One magical question and one simple formula are the ultimate measures of customer satisfaction and the ultimate predictors of future success (the so-called Ultimate Question and the Net Promoter® Score). The Net Promoter® Score (NPS®) is based on the Ultimate Question, according to its authors (Reichheld and Markey, Bain and Company ‘colleagues’). The Ultimate Question is, ‘How likely is it that you would recommend (this product, service, company) to a colleague or friend?’ Various sources slightly modify the question, but this is the basic form of the question itself. The answer scale is an 11-point, 0-to-10 scale, with the 10 defined as ‘Extremely Likely’ to recommend, and 0 defined as ‘Not At All Likely.’ The intent of the question itself is conceptually sound, and who can disagree with the proposition that it’s a good thing to have lots of customers recommending your company? However, the Ultimate Question answer scale and the way the NPS® is calculated are seriously flawed.

The assertion that this recommendation question is the Ultimate Question, of course, is open to question. Recommendations are important in some product and service categories, but not important in other categories. If word-of-mouth recommendation is important in your category, then a recommendation question probably is a useful measure for your industry, but there are always other important metrics to consider (future purchase intent, actual satisfaction with the product or service, reactions to pricing, etc.). It is generally wise not to ‘hang your hat’ on one question. It’s too risky.

The Ultimate Question’s 11-point answer scale is:

This scale is seriously flawed. Since the individual numbers (the answer choices) are not labeled or defined, what does someone’s answer actually mean? Is an 8 rating or a 7 rating positive, neutral, or negative? Some people tend to give high ratings, while others tend to give low ratings, especially when the points on the answer scale are not precisely defined. The 11-point answer scale is vague and open to respondent interpretation, and is probably not the best set of answer choices. We’ll come back to this later.

The Net Promoter® Score is calculated from the answers to the 11-point scale. The 10 and 9 ratings are grouped together and called Promoters. The 8 and 7 ratings are not counted at all, and those who choose 8 or 7 are called Passives. Those who give a rating of 6 or below are called Detractors. The definition of these three categories appears to be arbitrary.

The Net Promoter® Score is calculated by the following formula:

Percentage who answer a 10 or 9 (Promoters) minus the percentage who answer a 6, 5, 4, 3, 2, 1, or 0 (Detractors) = Net Promoter® Score.

Or simply, percentage Promoters minus percentage Detractors = Net Promoter® Score.

The Net Promoter® Score formula is flawed and imprecise because of lost information. Here’s how the NPS® loses information:

1. The NPS® counts a 10 answer and a 9 answer as equal, as the same thing. Isn’t a 10 better than a 9? This information (that a 10 is better than a 9) is lost in the formula.
2. If someone answers an 8 or a 7, the answer simply disappears; it’s not in the formula. So all of the information in an 8 or a 7 answer is lost.
3. The NPS® counts a 6 answer the same as a 0 answer, a 5 answer the same as a 0 answer, a 4 answer the same as a 0 answer, and so on. Isn’t a 6 answer much better than a 0 answer? So most of the information in answers 6, 5, 4, 3, 2, and 1 is lost in the NPS® formula.

In effect, the NPS® converts a very sensitive 11-point answer scale into a crude 2-point scale that loses much of the information contained in the original answers.

A far better measure than the Net Promoter® Score is a simple average of the answers to the 11-point scale, where a 10 answer counts as a 10, a 9 answer counts as a 9, an 8 answer counts as an 8 and so on down to 0. This results in an average score somewhere between 0 and 10 that contains all of the information in the original answer scale. No information is lost!

A final consideration is variation in the frequency of customer recommendations across industries and product categories. Someone may recommend a car dealership or a restaurant or a golf course (high-interest categories) but not mention a drugstore, gas station, bank, or funeral home (low-interest categories). Customer recommendations might not be an important factor in your industry. A sound strategy is to tailor the customer experience questions to your industry, your product/service, and to your business goals. Use multiple questions that measure customer experience from different perspectives. Don’t buy into the illusion of universal truth or an ‘ultimate question.’ Don’t fall for simple answers to complex questions.

Employee and executive compensation should be directly linked to customer satisfaction ratings. The essence of this myth is that employee compensation should go up as customer satisfaction ratings go up, to ensure a company’s success. This is generally not a good idea. It provides incentives for everyone to ‘game’ the system and try to cheat on their customer satisfaction surveys. (How many times have you had a sales or service person ask you to give them top ratings before you were surveyed?). Rarely are enough surveys completed ‘per person’ to provide statistically reliable data at the individual level. Often a better strategy is to base compensation on the judgments of on-the-ground supervisors, managers, and coworkers. Customer satisfaction ratings could be a factor in compensation reviews (if sample sizes are sufficient), but only a small factor.

Customer satisfaction scores are all-important, the ultimate determinant of a store’s, an employee’s, or a dealer’s evaluation by senior management. Many senior executives are not aware that satisfaction scores vary by different regions within the U.S., vary by the demographic makeup of a retailer’s customers, vary by age of the store, etc. These variances make it very difficult to compare the satisfaction scores from one store to the next, or across different parts of the country. Few companies use sophisticated modeling to adjust for all of these natural variances to ‘level the playing field.’

It’s all about the ‘customer experience,’ as though the customer experience is a magical, ethereal, transcendental idea. Duh? What is so revolutionary about the notion of understanding customers? Haven’t good companies been providing good customer experience since (and before) the coming of the Industrial Revolution? Isn’t marketing research about understanding how customers think, feel, behave, and react—at all of the ‘touchpoints’? Might ‘customer experience’ just be a new name for marketing research, or a new name for ‘customer satisfaction,’ rather than a revolutionary new concept?

The key to success is ‘raving fans’ or ‘super fans,’ not just satisfied customers. The idea of ‘raving fans’ or ‘super fans’ is catchy and cute, and one can easily imagine customers with painted faces dancing in the streets and singing the virtues of a company’s products or services. A very nice fantasy. Hype and hoopla are fun. Raving fans might be a possibility for a few product categories, but the majority of products and services don’t lend themselves to the arousal of extremes of passion.

Key Questions

Okay, so it’s easy to poke holes in the rhetoric surrounding customer satisfaction and customer experience measurement, but what are the serious questions to think about as you contemplate a customer satisfaction monitoring program?

• What are the objectives and goals of a customer satisfaction or customer loyalty program? Is such a program likely to achieve its goals?
• What are the relevant questions—the metrics—that make sense for your industry and your company?
• How much measurement is enough? Are surveys once or twice a year sufficient to accomplish your goals? Not every company needs to continuously measure customer satisfaction.
• How do you interpret the results? Can the results be organized, weighted, normalized, analyzed, etc., so that the ratings are unbiased and fair?
• Is a customer satisfaction program really worth the money it will cost?

Best Practices

Let’s assume you have done your homework and answered the above questions honestly, and you feel that a customer satisfaction and loyalty program makes marketing and economic sense for your company. You have clearly defined the objectives for the program. What are the best practices to help ensure that your company gets the most out of its customer satisfaction and customer loyalty programs?

Qualitative Explorations. Always begin with some type of qualitative research (focus groups, depth interviews, online forums, ethnography) to better understand your customers, their perceptions, their experiences, their language, their fears, and their issues and concerns. If you don’t really understand your customers’ feelings, knowledge levels, and perceptions, and don’t understand how customers interact with your company (at all the touchpoints), it’s almost impossible to design a good customer satisfaction or customer experience monitoring program. The qualitative research will make it possible to design a relevant and valid survey questionnaire. In fact, the qualitative research should be repeated every two or three years as a safety check to ensure that your satisfaction monitoring program remains relevant and on track. Another ‘best practice’ is to have qualitative research be an ongoing part of the customer experience program. The goal is to investigate each touchpoint through qualitative research, so that over a two- or three-year cycle all of the touchpoints are examined to identify improvement opportunities.

Sampling. Who should you survey? If you only talk to your current customers, you may be overlooking a sea of dissatisfied former customers. Generally, a best practice is to focus your surveys on your current customers, but include cells of infrequent customers and lapsed or former customers. If you conduct your surveys as visits or transactions occur, your sample will tend to be biased toward heavy users. This can be a good thing, since heavy users account for a large share of your business, but it can overstate your customer satisfaction scores because you are conducting a high share of surveys among your most loyal customers. It’s like ‘preaching to the choir.’

In most cases your employees, dealers, and agents should be surveyed as well. They represent your company and your brands. If they are unhappy or misinformed, they can quickly undermine any customer satisfaction program. Sampling must be carefully conceived and controlled to achieve optimum results.

Questionnaire Design. The first rule is ‘do no harm.’ That is, your attempts to measure customer satisfaction should not lower your customers’ satisfaction. This means that questionnaires should be simple, concise, and relevant. Use very simple rating scales (2-point, 3-point, 4-point scales). Short word-defined scales (e.g., excellent, good, fair, poor) are easy for customers to answer, and the results are easy to explain to executives and employees. Moreover, short scales work well on PCs, tablet computers, and smartphones. Long, complicated scales should be avoided. The questionnaire should almost always begin with an open-ended question, to give the customer a chance to tell his/her story. An opening question might be:

Please tell us about your recent experience of buying a new Lexus from our dealer in north Denver. The open-ended question gives the customer the opportunity to explain and complain; it communicates that you are really interested in the customer and his or her experiences; it conveys that your company is really listening. Then you can ask rating questions about various aspects of the customer’s experience, but keep these few in number. Most satisfaction questionnaires are much too long. If you want to include a recommendation question, we would recommend something similar to the following (a restaurant example, but remember the exact wording must be tailored to your industry, company, and situation):

Based on your last visit to the Burger Jubilee store at 105 North Avenue in Phoenix, how likely are you to recommend this Burger Jubilee store to your friends and acquaintances?

A. Would definitely recommend this Burger Jubilee store to friends and acquaintances
B. Would probably recommend this Burger Jubilee store to friends and acquaintances
C. Would probably not mention this Burger Jubilee store to friends and acquaintances
D. Would tell friends and acquaintances not to visit this Burger Jubilee store

And to calculate the Recommendation Score the following formula would be used:

(Percentage who answer A plus one-half of percentage who answer B) minus the percentage who answer D = Recommendation Score

This question, its well-defined answer choices, and the Recommendation Score formula will give you a much more precise measure of the net influence of customer recommendations than the Ultimate Question and the Net Promoter® Score.

The final customer satisfaction questionnaire should always be pretested with 20 to 30 depth interviews. It’s also best practice to do 500 or 1,000 surveys, as another type of pretest, and factor analyze the answers. Often this will allow you to shorten the questionnaire by eliminating redundant questions or attributes.

Survey Data Collection. The great challenge is to get customers to respond, since most are bombarded daily or weekly with poorly designed satisfaction surveys. Strive to avoid interviewing the same customers over and over again. Offering customers some type of discount or credit on future purchases or transactions (if they respond) is often sufficient to stimulate response. If the product or service category is a high-interest one (e.g., automobiles, smartphones, golfing equipment, etc.), you can often achieve high response rates without any incentives.

The method of data collection should be tailored to the customer base. Mail surveys work extremely well for automotive purchases, for instance. Telephone surveys are widely used because interviewers add a personal touch and can ensure that any problems are immediately addressed by the correct person or department. Online surveys are probably the most widely used survey method for customer satisfaction. It’s often the least expensive—if customers’ email addresses are systematically collected.

Smartphones and tablet computers increasingly offer new ways to survey customers. Complaint-tracking or problem-tracking (types of customer satisfaction research) can be conducted via QR codes, IVR (interactive voice response), or survey invitations on invoices, sales receipts, paper coffee cups, etc.

Other Data Streams. Survey-based customer satisfaction data is a solid, scientific foundation, but it’s not the only source of customer information. Companies have data flowing in from sales organizations, customer complaint letters, consumer affairs departments, CRM (customer relationship management) comments, social media chatter, etc. Think creatively about how all of these data sources can be tapped to create one integrated data set. Each stream of customer information is another ‘measuring stick’ to evaluate customer experience and satisfaction. All of this data should be analyzed at the same time, side by side. If all of the data are telling the same story, we can be more confident in the conclusions. If different data streams are telling different stories, then we must dig deeper to understand the differences.

Analysis. This is where most customer satisfaction measurement systems fail. If you are comparing different stores, or different geographical areas, you should normalize the survey results to make the comparisons fair and accurate. If you are looking at results for a particular store or division over time, you must be wary of seasonal variations and economic-cycle variations. When the weather is very hot, or very cold, customers might tend to be more irritable. During the Christmas season everyone tends to be emotionally stressed, and satisfaction scores can go down. The use of advanced statistical methods is essential to level the playing field.

A best practice is to build relevant and useful mathematical models that compensate for data biases and distortions, link together important measurements in sophisticated ways, and derive simple composite results that everyone, even senior executives, can understand and use. Use the Recommendation Score rather than the Net Promoter® Score, if recommendations are important in your business.

Benchmarking is important. You should collect customer satisfaction data for your major competitors, if possible, so that you have a relevant measuring stick to analyze your own satisfaction data. This is not easy to do, since you rarely have equal access to your competitors’ customers. A best practice is to conduct a benchmarking study every two or three years (using identical methods for all companies), so that you really know how your services and your employees stack up against the competition.

Marketing Strategy and Positioning. Customer satisfaction and loyalty do not exist in a vacuum. A company’s marketing, positioning, and advertising can have powerful effects on customers’ perceptions, feelings, and satisfaction levels. For example, if your Lexus breaks down, you might think it’s a rare exception because of the Japanese manufacturer’s high-quality reputation; but if your Chevrolet breaks down, you could think it’s just another example of sloppy American manufacturing. It’s really important to align a company’s positioning and marketing strategy with customer satisfaction strategies, so that all these elements are working together synergistically.