Is Your Customer Feedback Biased?

This post is an issue of the Savio newsletter. To get newsletters like this straight to your inbox, subscribe below.

Most of us collect feedback about our customers (satisfaction scores, survey results, feature requests, etc.) because we hope it will tell us something about our customers.

But what if the information we collect isn’t accurate? It turns out that it’s quite difficult to get unbiased feedback from customers.

But there are some things we can do to reduce the biases that affect both how our customers report on their experience of us and how we make product decisions.

Here are some articles we’ve read recently that tackle the issue of bias in our customer feedback and satisfaction metrics—and what we can do to improve our information quality.

(Wrote an article useful to PMs? Hit reply, send it to me, and I’ll try to include it in a future newsletter 🙏 )

Insights from around the web 🔎

Positivity Bias in Customer Satisfaction Ratings by Kunwoo Park, Meeyoung Cha, and Eunhee Rhim. Do you collect customer satisfaction ratings? Then you know that not all your customers provide ratings. In this paper, the researchers wanted to find out how those customers that didn’t respond would rate customer satisfaction. They found that the majority of non-respondents would likely be dissatisfied with the service. What that means for you: Your customers’ true satisfaction is probably lower than your stats suggest. Read it on ACM’s digital library →

There Is No Single Best Measure of Your Customers by Thorsten Wiesel, Peter C. Verhoef, and Evert de Haan. Speaking of customer satisfaction, what’s the best way to measure it? Customer satisfaction scores? Net promoter score? Customer effort score? These authors present evidence to show that none are necessarily the best. The takeaway: stay away from single customer metrics; instead, use more complex and nuanced metrics for predicting customer behaviour. Read it on Harvard Business Review →

The Behavioral Scientists’ Guide to Better Product Decisions by Hope Gurion. “As a product leader, I believe that there are two key ingredients for achieving positive outcomes: Truth and Math.” What an opening line! In this article Hope walks us through the biases in how we make decisions, how those apply to product decisions, and how to build systems to minimize our biases. Super valuable. Read it on Medium →

Potent quotables 💬

From Jono Bacon on Twitter: “Here is an example of [Mcdonald's] introducing unnecessary survey bias. This will significantly skew the results due to the reward for a specific outcome. This is not how we get better: reward people for constructive, comprehensive feedback, not merely the result you want to see.”

The receipt in the image says, “Tell us you are HIGHLY SATISFIED with your visit and receive a BUY ONE GET ONE FREE…”

Our take: Jono’s absolutely right—this survey demonstrates a fundamental misunderstanding of the reason we collect customer feedback. We collect feedback to understand what our customers think, not as a way to hear nice things. The more accurate and unbiased your feedback is, the better.

Helpful content from Savio 🦾

Why we launched a contrarian version of public voting boards by Kareem Mayan. Voting boards have been a popular way to crowdsource feature requests and product ideas since 2008. But the way most voting boards are built, they can actually bias feedback. Here’s why, and how we built our voting board with features that minimize distortion so your feedback data more closely represents what your customers and users actually want.

Minimize bias from your feature voting board →

Here’s to getting accurate feedback data from your customers 🙌

Last Updated: 2023-06-02

Want more articles like this?

Product Leaders from Slack, Zapier, and Appcues read our newsletter to delight customers, lower churn, and grow revenue.

Send me more like this

Unsubscribe anytime.