User Research Gone Astray

User Research Gone Astray

The Building Windows 8 blog offers some fascinating insight into Microsoft’s research and design processes. Take the recent post on improving Windows Explorer.

It begins with a discussion on how Windows Explorer is used today. Through telemetry data — “based on hundreds of millions of individuals opting in to provide anonymous data about product usage” — Microsoft finds that the top 10 commands make up 81.8% of use, and that these commands (e.g., paste, properties, copy) are primarily accessed through the contextual menu.

The development team combines this with customer feedback on the top requested features, and uses this to justify the changes from Windows 7…

…to the design below.

It seems that essentially, every single command that customers have used or requested has been moved into a ribbon or wedged into some corner of the chrome. And many are rightfully lambasting it.

But, you might protest, they did research! With users! And got a ton of data! So how did it go wrong?

The IROP Keyboard

By focusing user research on low-level operations with the old system, and using that as the starting point for the redesign, you end up merely resizing, rearranging, and removing parts of the interface. You don’t make the quantum leap, and you sometimes make things worse.

I’m reminded of a recent paper by Andreas Zeller, Thomas Zimmerman, and Christian Bird (the last two authors from Microsoft Research, ironically) titled Failure is a Four-Letter Word: A Satire in Empirical Research.

In the paper, the authors collect keystroke-level data in Eclipse and correlate it with programmers’ errors. They find this data to be an excellent predictor, with the letters “i”, “r”, “o”, and “p” guilty of the strongest correlations. Based on these findings, they come up with a cheeky solution for reducing programmers’ errors:

The IROP keyboard addresses the data, quite directly. But it obviously misses the deeper problems.

From Actions to Activities

In the vocabulary of Donald Norman, operations combine to make actions, which make tasks, which make activities. According to this scheme, Microsoft has with their data looked at actions. The key is to start higher up this hierarchy, considering the activities users are engaged in and the goals they’re trying to achieve.

Windows Explorer is gradually being sunset, so they get some slack here. But I sure hope they aren’t being steered wrong elsewhere by their user research.