In the past year, I led the effort in a few UX research initiatives, which involve a bunch of remote usability tests with technical users for enterprise applications. Although my focuses on each testing round are different, there are some common best practices I’d like to share.

1. Always define the scope first

At medium-to-large size companies, UX researchers will help recruit participants and facilitate the sessions, and it is necessary that the designer walks through the prototype and shares assumptions with the researcher first. The interaction flow within a business application can sometimes be very complicated, and it will be beneficial if you can annotate the “pathway” for UX researchers. An example pathway can be:

Select [ItemA] and [ItemB] -> Select [Action1] from [Actions] dropdown.

Meanwhile, if you plan to invite users who have different knowledge levels with your application, you may have different assumptions for the groups, and the researchers need to understand where to probe when testing with them. When defining assumptions, you can use a structure like this:

We believe [feature] is important for [end-users] because it addresses [user need]. However, [end-users] may have difficulty in [pain point].

At smaller organizations, UX designers may need to undertake both design and research responsibilities. Using the above template as a checklist can also help you reduce unconscious bias when ideating interview questions and conducting testing sessions.

2. Define the level of interaction you expect participants have

If you are the person who facilitates the remote testing sessions, you need to specify the phone/screen share tool you plan to use and notify the participants in advance. Before diving into the interviews, you also need to define how you expect the participants to interact with you.

Do you expect the participants to open the camera so that you can observe their reactions? Do you want to record the conversation (audio & texts only)? These are requirements that you need to always communicate with the participants during the warm-up.

What kind of usability method do you want to use? For instance, if you expect the participants to apply the think-aloud protocol, then you need to specify and explain it before sharing the prototype. You also need to be aware of the pros and cons of using this protocol — depending on your purposes, “think aloud” may not always be the best method to use.

3. Gather qualitative insights through open-ended questions

When running usability tests for an e-commerce website 3 years ago, I remember we used task completion rate and time spent on task as two major UX metrics. However, it’s not the case for business applications, especially when the end-users have extensive knowledge within the application — even though some action buttons are hidden in default, the power users can still achieve the assigned tasks without additional help, simply because they have been using the product in daily work for years.

Task completion rate is a fundamental usability test metric. However, it CANNOT really provide insight when testing with power users.

When testing with power users, it’s more important to understand their mental models using two methods:

• Observe how they perform the tasks: One thing I learned is that behavioral and verbal feedback from users are equally important. For example, when viewing and examining the following table, the users may move the mouse over the first column to see if they can open a hyperlink, when they realize it doesn’t work, they hover the mouse next to the texts, and then click the dropdown icon that just appeared. Even though they still completed the task, their behavioral traits exposed usability issues that UX designers should be aware of.

• Ask open-ended questions and learn how the experience matches their expectations: A good open-ended question can be“How would you lay out information shown in this panel, so that it can assist you to make decisions at this point of your workflow?” It’ll help you gather more insightful feedback than simply asking “Do you like the information layout shown in this panel?”

4. Use pilot tests as opportunities to fill in the gaps

You can consider pilot tests as rehearsals to help you gain the confidence to move forward with the main study. If possible, you can also invite a subject-matter expert and another designer to the pilot test. They can help you realize that the phrasing of some questions has already given away the answer; or you originally thought you clearly explained a task, yet the subject-matter expert pointed out you could use some industry jargon in your scripts.

Getting their perspectives can help you find gaps in the task flow, and may even help you find an unexpected “A-ha” moment that may reshape your strategic approach. 💡