Overview

In the wide world of UX design, specifically in UXR (user experience research), there are a plethora of ways, activities, and lines of questioning that you can use to get to know your users better.

While there are many approaches that work incredibly well, there is one particular type of question that I strongly admonish every designer to steer well clear of.

To that end, while I’m not going to cover the entirety of UXR here (I already did in the linked article above in detail), what I AM going to do is share with you one of the worst questions you can possibly ask when conducting UX research, and what to ask instead.

An innocent question with horrible outcomes

What starts off as an innocent question almost always leads to horrendous outcomes in terms of implemented design specifications.

This question almost always takes the form of:

“Would you ______________________?”

DO NOT ASK QUESTIONS LIKE THIS, EVER! Burn this into your brain: “would you ___________” questions are categorically off limits in UXR.

Why? Because users are notoriously bad at predicting future behavior and/or preferences, as they exist outside of the context that drives the behavior in the first place.

Don’t believe me? It’s cool, I was skeptical too. I gotchu.

Let’s talk proof.

Let’s check out a case study where listening to users’ responses from a “would you _______________,” question cost Walmart over a billion dollars in lost revenue.

Why this question is so problematic

Back in ’09, Walmart decided it wanted to try and compete more directly with Target, so it surveyed its customers and asked them:

“Would you like Walmart aisles to be less cluttered?”

Simple question. Innocent question. No big deal, right?

Customers responded with more or less, “yeah, sure, that sounds good.”

So Walmart did. They made the aisles less cluttered. They came up with a solution, asked customers if they thought it was a good idea, they implemented it, and everything worked out great.

→ Except it didn’t.

“Sales went down. Way down. I mean waaaaaay down. I’m talking, from the beginning of that project until today, Walmart has lost over a billion dollars in sales. (Yes, billion with a “b”.) It’s actually closer to two billion dollars of sales they missed out on, and maybe more.” — Phil Terry, Good Experience

What does this tell us? Two things:

1. Users cannot accurately predict future behavior or preferences

It is a capital offense to assume that users are good at predicting what they will do in the future through routine and/or consistency of identity alone.

People are flexible, malleable, and subject to change on a minute-by-minute, hour-by-hour basis. What a person wants and needs changes in response to their environment and contextual stimuli.

2. Users cannot answer questions about future circumstances

Similarly, users cannot accurately answer questions about future or hypothetical circumstances, as their data is limited solely to what they have already experienced, and any attempt at doing so represents pure conjecture.

Future questions strip out experiential context and force users to attempt to render the situation purely hypothetically.

In short, they would have no idea, and neither would you.

A much better way to address this

So what can you do? How can we address the task of figuring out what users need and want without asking questions that prompt users to prognosticate the future?

The answer is actually deceptively simple:

Use “what” and “how” questions, geared towards things users have already experienced.

Let me give you a BAD example

• “Would you buy a sandwich with pickles on it if you were hungry?”
• “Yes, if I was hungry I’d buy a sandwich with pickles on it.”
• Fast-forward a couple years.
• “Hey, are you hungry?”
• “Yeah, starving, why?”
• “I have a chicken sandwich with pickles to sell you, pretty tasty, only $9.99 a month, does exactly what you want, we just got through our Series C and our investors are hounding us for ROI, please buy the sandwich that our research said you would buy.”
• “Yeah, I don’t really want a chicken sandwich with pickles, but thanks.”

You see the problem here? This is a visceral, bad example of something that happens constantly; where companies will ask users to predict their future behavior or preferences without directly observing them do it, or measuring past behavior/preferences as an indicator.

Let me give you a MUCH BETTER example

• “How do you feel about pickles on sandwiches?”
• “They’re okay, I mean they’re not my favorite but I’ll eat them.”
• “Speaking of, tell me about your experience with sandwich shops, what has that been like for you?”
• “You know I hate how whenever you go in the line is always long as hell and slow, and then you can’t just buy a pre-made sandwich, they gotta make it in front of you and…”

BAM. Soooooo much better. Nice, open-ended, non-biasing question that allow the user to talk about what THEY want to talk about within the context of your line of questioning.

You can also use this technique to steer users back to your line of questioning without biasing them further in the process.

• “Yeah, I really like mayonnaise on my sandwiches, my local market had a great deal on that the other day!”
• “Really? What about the deal was so good? Was it a sandwich-based deal or another type of deal entirely?”

Again, see what we’re doing here: we are actively steering the user towards our sandwich-based questions without biasing them one way or another; we’re attempting to figure out not what goes on the sandwich that they’ll buy but what they value in a sandwich and sandwich experience on a personal level.

Most of the time, a user cohort will value similar things, so it makes sense that if you can get to understand the emotional, utility, and convenience values they’re looking for in a given solution, WITHIN a given usage context, you’ve got a much better shot as designing a winning product.

The bottom line

So what does this mean for you as a designer?

• Never ask “would you _______________,” questions to your users, as they prompt users to attempt to predict future behavior/preferences, which they can’t reliably do.
• Instead, use “what” and “how” questions, geared towards things users have already experienced, in order to get the information you’re looking for.

Past behavior and preferences are the greatest predictor of user behavior in the future, so its imperative that we go off of what the user has done, preferred, or experienced before, rather than what they think they might do, prefer, or experience in a hypothetical future.

This allows you to garner a much clearer picture of your users’ real past experiences, which in turn define their expectations within the domain of your line of questioning.