Is Talking to Users Enough?

Recently a friend of mine formed a startup with a goal of making the One-On-One Meeting an easier and more valuable process. Their aim was to help managers and their direct reports easier handle the goals and issues that arise from these sessions. The assumption they made was that their customers (primarily managers of developers and testers in IT companies) would pay a subscription fee to use their application, which would allow them to avoid manually recording, tracking and sharing all of these in spreadsheets.

My friend (an experienced and passionate Agilist) first decided to talk to potential customers to test the viability of the idea and to determine if they had achieved a Problem/Solution fit. As it happened, two of the potential customers she interviewed work for a client of mine. After the interview, my friend came away positive that her product would solve a problem for these two potential customers at least, and that they’d pay for it. So far, so good right?

Not so fast... Back in the office, I chatted to my two co-workers and it quickly became clear that although they thought the product was a good idea, they wouldn’t pay any money for it (which was unfortunate as the business model for the startup was dependent on a subscription fee!). Obviously, there was a disconnect between what was discussed, interpreted and reality, but why?

Talking to your users is crucial, but it is not enough

When developing a new product or service, your goal must be to build as little as possible to validate your idea and find out if you've got something worth actually developing. Getting to know your intended customers and their problems is an important part of that process. There are however some caveats and limitations of talking to users about your idea that need to be taken into account, for example:

The Observer Effect

The Agile community is sometimes accused of bastardising concepts from physics and the other sciences, so I know I’m in good company here! This states that the simple act of observation alters the state of the system that is being measured. In the example above, talking to your interviewees can affect how they perceive the problem and the fit of your solution. They may actually subconsciously alter their responses to match what they think you would like to hear! Very gratifying, but this leads to ultimately inaccurate information...

Confirmation Bias

It’s only human nature; it’s your idea, your pet project. It’s very hard to separate yourself from the belief that your product or feature truly is the solution to everybody’s problems. It’s incredibly difficult to remain impartial when discussing your idea and there’s a high probability that this will affect your interpretation of the results.

User Diversity

With any product, your user group is likely to be diverse; differing in their role, background, education, needs and more. It’s impossible to cover all of these during the interview process. This means that any conclusions drawn from your interviews are far from conclusive of how the total population of the target user group will behave.

My point is that speaking to your users is an important part of discovery, but be careful and deliberate in how you go about this process, and how you interpret the data. Justin Wilcox has some great insights on how to better go about interviewing customers here. His two rules of customer interviews are:

#1. Do not talk about your idea

Talking about your idea skews how both you and your interviewee will behave, and will compromise the validity of the data you gather. Instead discuss the customer and their problems.

#2. Do not ask about the future

Don't ask the interviewee to predict the future, or how they may hypothetically behave. Speculation is not reliable data, and what you're doing is generally a thinly veiled version of #1

To put this in a format familiar to us all:

Ultimately, what you emerge with from a successful user discovery session is still an assumption, albeit hopefully a better one than you began with. The true measure of course of whether there is a fit is not “would” they pay money for your product, but “will” they pay money for it, right now?! Even better, will someone who doesn’t know they’re part of a test group pay to use your product or service?!

Your assumption (like any other) still needs to be validated, as cheaply as possible and with actual users who (depending on your business model) will pay actual money for it. This is where the concept of MVP and the Build-Measure-Learn loop comes into play, but that's a subject for another post!

Happy innovating!

Read More

1. Steve Blank has a huge amount of material available on Customer Development and Innovation on his blog


2. A Lean Approach to Product Validation - Smashing Magazine


3. Of course, Eric Ries' The Lean Startup


4. How Justin Wilcox interviews customers, on Customer Development Labs