The car saleswoman leaned over the desk, placed a sheet of paper down in front of us. “This is a version of the customer satisfaction survey they’re going to send you next week in the mail.” It was a garden-variety survey about the car purchasing process, with a five-point scale that ranged from Excellent (5) to Poor (1). She plucked a pen from the desk before her, then drew a hard blue line down the page between the 4 and the 5.
“If I’m rated anything less than a five,” she said, “then I’m not doing a good job. It’s either a five or I’ve failed.” She continued to describe how even one 3 or 4 in a month could lead to her not being considered a top salesperson. Then she wouldn’t be able to take her family to Hawaii or the Bahamas at the end of the year, courtesy of the car manufacturer, as a thank you for her performance.
Talking about the whole situation after dinner, my wife Mary said:
Buying a car sucks. Everyone knows it. There’s no way in hell this survey thing is working. If they’re all getting fives and the company says less than five is a failure, then why does buying a car still suck? These surveys send employees to Hawaii—they aren’t actually influencing the overall customer experience.
Surveys can be useful instruments for gathering data from people, but survey questions can be bent or manipulated in ways that destroy their benefit for everyone. In the end, is anyone actually using this data to improve customer service? Or are they just using it to punish service providers?
Rather than relying on the illusory difference between a 4.5 and a 4.7* for quality service, I’m more interested in how someone answers the following simple question: “Was this service great? Yes or no, then tell us why.” In the case of car surveys, this type of open-ended question is one of the few places post-purchase where direct service improvements are solicited. Matt Jones at Edumunds says “at the end of Customer Satisfaction Index (CSI) surveys, there is a comment section for the car shopper to address any concerns that may have come up while doing the deal. These comments do not affect the overall scoring of the salesperson. If a car shopper thought the music was too loud in the dealership, for example, saying that in the survey comment would likely be a better option than giving the salesperson an 8.”
So next time you think about using a graduated scale in your survey, know why you’re using it, what biases those delivering the survey may introduce, and whether there might be a better research instrument to achieve a more relevant result.
* We see this in services like Uber, where if a driver's rating dips below 4.X stars they may not be able to participate in the service. Absolute perfection required.