The Benefits of an Open User Research Practice

I never really loved mathematics. I am much more of a big picture person than a tiny detail person. But I usually did ok in maths tests because you got marks not just for the answer but for showing all the thinking you did to get there. I may not always get the answer right, usually as a result of a simple mistake along the way, but you can see how I am thinking and understand where to intervene to correct. We both learn.

I apply the same approach to research practice, especially when working with teams who may not have a particularly strong understanding of how and why we do things as we do. An open research practice has multiple benefits including:

• learning about how and why you think about and make decisions and actions at each stage
• understanding what tradeoffs are being made and the impact this has (there are always tradeoffs)
• understanding how we move from data to insight and deeply understanding and trusting what we have learned.

Openness in research requires the willingness to adapt, to not always be right and perfect, to go slower than you want to (and often far slower than people expect) and as a result requires decent amount of bravery.

There are three crucial stages for openness in research practice:

• study design
• fieldwork
• analysis and synthesis

Openness in study design

Being open in the study design phase means bringing your team into the process about considering who we want to talk to (and, as a result who we choose not to include), and what we want to talk with the about. In particular, the consideration of what kinds of differences matter in our audience base is an important one to have and one that thinking about research recruitment can help to facilitate.

Openness in fieldwork

Being open during fieldwork refers to a researcher’s willingness to have team mates observe the research as it happens. There are many different ways that you can enable this, and different levels of interactivity that your team might have with the participant during the study. Being comfortable with having your team observe as you conduct research can be really challenging for researchers at first. Once this becomes standard practice though, it quickly becomes an essential part of our practice and help us to demonstrate the differences between the research questions we want to answer and the questions we need to ask participants in order to answer those questions.

Many researchers are concerned that participants will observe one session and run off to change the entire product based on a single data point – although this is a commonly voiced concern it is usually easily managed by clearly setting expectations that everyone on the team is required to observe at least two sessions before being allowed to participate in the analysis process (from which the findings emerge). We often use UIE’s Exposure Hours requirement for at least 2hrs of observation every 6 weeks as a metric that helps encourage team mates to experience more than a single session in any one research study.

Openness in analysis and field work

While giving team mates the opportunity to observe their customers and users first hand has obvious benefits, allowing them to participate in the analysis process is arguably even more important. This is where we truly pull back the curtain and show the hardest work of research which is making sense of all the stories we have heard and things we have observed.

Robust analysis and synthesis is probably one of the most overlooked aspects of the research process – all too often we see examples of people observing a number of sessions, taking a few bullet point notes et voila – the findings immediately emerge.

If only it were really that simple. Analysis and synthesis is hard, time consuming work when done properly. Doing it properly is essential if you want to do the work required to rid yourselves of as many of those annoying cognitive biases as possible – in particular the confirmation bias and recency effect.

Allowing and encouraging team mates to participate in research analysis gives them an opportunity to get much closer to more of the data, but it also helps them to understand the way that we process that data in order to make sense of it and draw conclusions. It allows them to challenge the ways we are forming narratives about what we believe that data means and demonstrates the traceability of those claims back to the original source data.

This blog post and video describe how I’ve done collaborative analysis successfully with teams.

Open research is challenging but worthwhile

It is beyond dispute that working in an open way – research as a team sport – is slower and more painful for researchers than putting our heads down and getting through the work alone. It is, on the surface, less efficient and more annoying. Nonetheless, if you want to grow understanding and respect for the research craft in your organisation, it is very much worth the overhead to take the time and effort to open up your processes to your team and invite them in to participate actively. Overtime, the increased Research IQ in your team will pay dividends and your ability to be impactful more efficiently will increase.

It is important to remember that the most important thing is not the time to deliver the report, but the impact our research has on our teams ability to make good decisions for our customers and our users. Let’s make sure we’re optimising for efficiency to  the right outcome.