Let’s not confound our jobs

When usability testing

DON’T
- “The goal of the test is to find problems in the design.”
- “We want to see whether it is understandable and usable.”
- “We’re asking people to try using a Web site we’re working on so we can see whether it works as intended.”

DO
- “Our goal is to understand how people use this product”

Rationale

If we give away the goal of the research -that is to find the usability issues- participants will try extra hard to find us some. They sense the demands of the study and try to supply it. This would produce a lot of “false alarms” or fake results. This is also known as “Demand Characteristics bias”. It is bad, because we don’t want to spend our precious dev effort on fixing non-issues.

When I am briefing participants I don’t mention at all that we are out to find issues. I tell participants that we want to understand how people use the product, how they go about certain tasks, and how they are thinking while doing so… Now, this instruction creates the right Demand Characteristics: if participants want to be helpful, all they need to do is use the product and think aloud.

Let’s not confound our jobs

If you think about it, it’s the Researcher’s job to identify usability issues based on what the participant does or says. The participant’s job is simply to complete the tasks and think out loud. Let’s not mix the two.

Ethical considerations

Researchers must disclose the purpose of the study. I think if we say “We wanna understand how people use this product so we can better serve them” is a better way to disclose than “We are out to find problems.” The former is also the truth and the full truth, and it doesn’t create wrong demand characteristics.

Here are some more examples of instructions that create wrong demand characteristics.

DON’T
- “Your comments will help me understand the usability of the website.”
- “Our goal is to see how easy or difficult you find the site to use. Getting feedback from you really helps us with that.”
- “The purpose of doing this is to understand how we can make the website easier to use.”
- “I am a User Experience Researcher, I help the company to improve their products…”

Thanks for reading. I am a Sr UXR at SAP Emarsys. Follow my work on Twitter or at zombor.io