Spark23 Product Advisory
Product Advisory

Spark23 Blog

Observations and advice on product strategy, product management, innovation, and business culture.

Your survey data is bullshit – trash it and take these 5 steps

While waiting in the reception area of a shared office space yesterday morning, I was invited over by one of the three idle 20-something women sitting together at a table.  She asked me if I'd take a short survey – "it'll only take 3 or 4 minutes, and you'll get a $5 gift card to a coffee company."

I said, "Sure, no problem," and she handed me her tablet with the survey. 

A not very interactive survey on a rigid tablet 

A not very interactive survey on a rigid tablet 

I was immediately stumped on the first multiple choice question, which read, "How long have you been at this work space?"  

The correct answer would have been "about 2 minutes."  

But the choices all implied I was working there full time for varying lengths of months.

So, I immediately told her, "It seems this survey isn't really for me – I don't work here, I'm just here for a meeting."

She sat back in her chair, a little surprised, and looked at each of her two colleagues before glancing back at the tablet and said, "Well, just pick the first option, which is less than 1 month."

OK...  so I did that, and then proceeded to answer the 3 or 4 other relatively meaningless demographic questions on that first page about what I do at the space.  

At this point, my data in their survey implies that I recently joined that workspace as a new company, my company is very small, and I've been there less than a month.  

How might their company act on this false information?

  • First, they'll assume that my small, new business represents some percentage of that entire workspace.
  • Second, they'll assume that this percentage of companies have just joined the space – and what a great opportunity it would be to get in quickly and develop a relationship with them.
  • Third, they'll hold a brainstorming session on "welcome to the space" campaigns, specifically targeted to me and my small business needs in this new space.
  • And so on...
Are you making big business decisions on bad data?

Are you making big business decisions on bad data?

I moved onto the next page of the survey which included questions that seemed very specific to the solutions their company might offer me – I quickly realized that the survey and the three women were representing a very large and well-known data & security company.

All of the questions were multiple choice and I answered all of them truthfully and honestly about my needs regarding data security, what percentage of my documents are digital vs. paper, how I manage paper, how important security is to my business, etc.

While I took the survey, the three women sat there politely, talked amongst themselves and fiddled with their tablets. 

Are these dudes your customers?

Are these dudes your customers?

I finished, got my gift card and walked away.

Then I noticed another person going over to the table to fill out the survey.  I looked around the space and I realized the only people that were filling out this survey were glassy-eyed engineering-types who were stretching their legs and getting some coffee.  

What do each of these folks know about their employer's data security needs?

I can save this data security company a ton of money right now – please go ahead and drag your survey results over to the Trash.  There, I have just saved you tens of thousands of dollars that you would have spent designing products, services and campaigns that were destined to fail because they were based on meaningless data contributed by people who are not your customers!

Ya, I'm pretty sure

Ya, I'm pretty sure

Here's a better idea – listen to your customers!

Here are the five steps I would have followed at this data security company to learn more about my customers in this workspace.

1. Create personas for your target customers that includes demographic information, assumptions about their problems and goals, and how they might benefit from your products and services.

2. Write out a script of questions you'd like to ask your customers to learn more about who they are and what they do in their company; about their problems and their goals; their expenses and their budget, and so on.

Note that unlike the aforementioned survey questions, you're not asking them how interested they might be in your products and services.  Instead, you're using these questions to validate your assumptions in the personas you put together.  Your goal is to uncover the truths and facts about your customers!

What an engaging and insightful discussion!

What an engaging and insightful discussion!

3. Send 2 people with notebooks, rather than 3 people with tablets. (more money saved!)

4. Ask volunteers for 10 minutes of their time for a $20 gift card, rather than 3-4 minutes of their time for a $5 gift card.  How much is quality information worth to you? 

Here's a pro tip – use a qualifier question to make sure your volunteer roughly fits into one of your target persona profiles.  If the volunteer doesn't fit any of your profiles, send them on their way – he/she is not your customer and you will just be wasting their time and yours! 

5. Use the script to conduct the interview.  One person conducts the interview, the other person takes notes and records responses.

The interviewer should take cues from the interviewee. Ask "why" follow up questions to get to the root of their responses.  The note taker should pay special attention to the interviewee's affect and body language – these are vital signals that reflect how interested (or not!) they are in your business.

The results are in

Now, when you get back to the office, instead of looking at numbers from bad survey data, you'll have pages and pages of rich qualitative data to sift through.  You now have three important takeaways that you would not have gained from your survey:

1. You are 100% confident you spoke to the right people.

2. You have collected vital information about how your target customer talks and thinks, which can be used in your products, services and marketing campaigns.

3. You not only know which problems and goals are important to your target customers, but because you were studying their behavior, you also have signals as to how important these problems and goals are to them!

Thank you for reading – please comment and share with others!

Alex Westner