5 mistakes we all make with product feedback

It rarely makes sense to take feedback from all users and it never makes sense to get it all at once.

At the outset of a new project, or especially if you’ve recently taken over a product, it’s tempting to survey all your users to appraise where things are. It’s usually a mistake. In fact there are five common mistakes that we see over and over. Intercom makes getting feedback extremely easy, and as a result, it’s easy to become a little trigger happy with the feedback requests. Here’s five quick fixes for product feedback:

1. Stop talking to “all users”

When you survey all your users together you ignore the specifics. You mix up yesterday’s sign-ups with life long customers. Those who used your product every day with those who log in just to update billing details. Those who only use one specific feature with those who use them all. It’s a mess.

Solution: There’s a much cleaner way to get much better feedback. Here’s some examples:

• If you want to improve your onboarding, only listen to people who recently signed up.
• If you want to improve a feature, only talk to those who use it.
• If you want to understand why people aren’t using a feature, only talk to those who don’t use it.
• If you want to find areas of concern, only talk to active users who use all your features.

2. Feedback should be on-going

The default approach to feedback is to solicit it on demand. But that means when you realise you need it you have to wait a week doing nothing while it comes in. To compensate for this you cast a very wide net, ask a lot of questions, and sit back. If you’re particularly naive you’ll act on each piece as it comes in, rather than waiting and analysing it in whole. The problem here is twofold: firstly you never have feedback to hand when you need it, but secondly you only hear about problems when you choose to ask about them. This means you’re blind to gradual degradation of your product.

Solution: Periodically check in with users. The simplest, yet still valuable, version of this is to ask users for feedback on day 30, 60, 120, 365, etc. This takes about 20 seconds to set-up in Intercom, and will pay for itself in a day or two. A slightly more advanced version would be to gather feature specific feedback based on usage. For example, if you have a calendar tool, you might ask someone for their thoughts on the first, twentieth and fiftieth time they use. As a user gets used to a product their feedback matures. The first usage feedback will explain what’s confusing, the twentieth will explain the frustrations, the fiftieth will explain the limitations.

3. Distinguish Free from Paying Feedback

Related to point 1, it’s easy to assume all requests are of equal value, regardless of the state of an account. This is roughly true within certain thresholds (e.g. $50->$500) but there’s a noticeable difference between the type of requests you get from free users and from paying users. Long term free users are only capable of giving you feedback on how to improve your free plan, which is rarely a focus for a business. Typically, free plans exist to draw customers in and upsell them. You can’t listen to hypothetical feedback: “I’ll upgrade if…”, “I’ll upgrade when…”. Espoused behaviour is rarely useful, learn from things that actually happened.

Solution:

• To improve your product for your paying customers, only talk to your paying customers.
• To learn what makes people upgrade from free, only talk to customers who upgraded from free.
• When you want to improve your free product, only talk to your free customers.

4. Don’t fall for the vocal minority

It’s often said the plural of anecdote is not data, but that does not mean anecdotal evidence is not useful. The plural of anecdote is a hypothesis, or narrative. Something that’s easily verifiable. So on a day when five users ask for a simpler event form in the calendar, you don’t assume five people represent all users and immediately kick off an “event simplification” project. First you should attempt to verify if these five users represent all users. You start talking to calendar users, and see what else comes up.

Solution: Treat every clustering of feedback that you see as a hypothesis, and then don’t build it, verify it. Once you verify that the pain is real, the next step is never “build the requested solution”, you have to go deeper. Which brings us to the last point.

5. Don’t assume users request the right feature

To paraphrase Confucious, when customers point to the moon, the naive product manager examines their finger. The faster horses tale is often used to justify not listening to customers, but that’s an epic way to miss the point. If a customer says they want a faster horse, what they’re actually telling you is that speed is a key requirement for transport. So you think about how you deliver that. In our previous example, our friend had five people complaining that her new event form was too complex. She could have lost weeks building a natural language input, or streamlining the UX of the form, but it turns out none of that would have helped. When she talked to all the calendar users, she quickly learned the pain came not from the form complexity, but from how often it had to be used. What actually solved the pain was recurring calendar events, and making events easy to duplicate.

Solution: Be aware that customer feature requests are a cocktail of their design skills, their knowledge of your product, and their understanding of their current pain point. They know nothing of your product vision, what features you’re currently working on, or what’s technically possible. This is why it’s essential to abstract a level or two above what’s requested, into something that makes sense to you, and benefits all your customers.

Of course it’s worth noting occasionally a feature request will be spot on. It will rhyme with every thing else and perfectly fit in the world the way you see it. On these occasions you can skip steps, namely verification, abstraction, and clustering and trust your gut. Your product intuition gives you wonderful shortcuts, so long as you’re still a true user of your product, and constantly in touch with the needs of your users.

But on every other occasion, talk to your customers, it makes you smarter.
______________________________________________________________________________________________
This is an edited version of a post originally posted at http://blog.intercom.io, by Des Traynor, Co-founder @intercom.You are free to re-edit and repost this in your own blog or other use under Creative Commons Attribution 3.0 License terms, by giving credit with a link to www.startupcommons.org and the original post.