Sometimes it feels like UX Research has become synonymous with surveys. They’re relatively cheap, easy and can reach large amounts of users. Sounds great, right?

Wrong. Surveys aren’t as easy to design as people assume. If you ask biased or leading questions, you cannot trust your results. My friend and former colleague,

wrote an excellent four-part series about this (links at the end).

Beyond possible biased results, you still may not get the full picture from a survey only. If you ask people how easy or difficult they find a task (or using your product) their answer may not be the same one you would draw if you observed them completing the task. This point can be illustrated with the exaggerated example of labour and delivery.

Labour and delivery

If you watched someone (without an epidural) go through labour and delivery you would probably observe they are in a great deal of pain. If you saw the same person shortly after birth they may seem pain-free and possibly overjoyed. This is because of the oxytocin released by the body and the fact that they (hopefully) have a healthy and loved baby (the halo effect). If you ask the parent how bad childbirth was after the fact, they may downplay the amount of pain they experienced at the moment.

Let’s take this example to the extreme, and say that we can only administer a survey and not observe any labouring and delivering people. We would determine that childbirth doesn’t hurt that much.

The same is true of user research. I have observed participants not complete tasks but still rate the task as ‘very easy’, 7 out of 7, on the SEQ*. I’ve also seen them struggle with every step of the process, only to state that a task was very easy.

*The Single Ease Question (SEQ) is a 7-point scale assessing how difficult or easy users find a task. It should be administered immediately after a user attempts the task in usability testing (Sauro, 2012).

Does this mean that people are lying?

Photo by Milk Chan on Unsplash

No. This discrepancy happens due to a combination of factors (including impression management whereby the participant wants you to think they found a task easy and knew what they were doing because they are smart and capable). Two factors in particular are:

1. The reflecting self sees things differently than what the experiencing self encountered at the time
2. The Dunning Kruger Effect in practice

The experiencing self versus the reflecting self

What one experiences at the time is referred to as the ‘experiencing self’ (Zajchowski, Schwab, and Dustin, 2016). Reflections on experiences after the fact, done by the ‘remembering self’, lose the nuance of in-the-moment understanding (i.e. the experiencing self during labour being in a great deal of pain vs the remembering self downplaying it).

The recency effect can cause a different interpretation of what is remembered, than what the experiencing self encountered at the time. That is, retrospective information from people who are using your product without issue, or who have completed a task, may minimise difficulties encountered. If they have persevered through difficulties they may not even recall that they experienced them at the time.

While Kahneman (2011) concludes the remembering self is the dominant means by which lives cultivate meaning, the experiencing self has implications for the technologies that are continued to be used (or abandoned) and for usability studies.

Dunning Kruger Effect

The Dunning-Kruger effect is often used to speak about people being ignorant of their own ignorance. More specifically it is a cognitive bias where people assess their abilities to be greater than they are. This is because people are not good at evaluating their own competency (Dunning, 2011).

Participants are assessing how easy a task was using the same ability and skills used to complete the task. If they think they finished the task successfully, it makes sense that they then rate it as ‘easy’.

You need to observe your users, not just survey them

These two factors highlight why we need to observe users interacting with our products, not just ask them to complete surveys. The experiences recalled may not be what was actually experienced at the time. Further, while they may tell you something was easy, if you have watched them struggle through the process you can identify possible usability issues and opportunities for improvement to reduce user frustrations (either those that are unexpressed or frustrations that a wider audience may face). Doing so will improve your product by reducing pain points and increasing usability.