What is data-driven design?

Simply put, data-driven design means making design decisions based on data you collect about how users interact with your product. According to InVision:

Data-driven design is about using information gleaned from both quantitative and qualitative sources to inform how you make decisions for a set of users.

Some common tools used to collect data include user surveys, A/B testing, site usage and analytics, consumer research, support logs, and discovery calls. By crafting your products in a way that cater to your users’ goals, preferences, and behaviors, it makes your products far more engaging — and successful.

While most data is quantitative and very objective, you can also collect qualitative data about your users’ behavior, feelings, and personal impressions.

A designer’s instinct

Back in the days of Mad Men, a designer’s gut instinct was glorified because it was difficult to measure the success of a design in progress. You often had to wait until it shipped to know if your idea was any good. Designers justified their value through their innate talent for creative ideas and artistic execution. Those whose instincts reliably produced success became rock stars.

In today’s data-driven world, that instinct is less necessary and holds less power. But make no mistake, there’s still a place for it.

Design instinct is a lot more than innate creative ability and cultural guesswork. It’s your wealth of experience. It’s familiarity with industry standards and best practices. You develop that instinct from trial and error — learning from your success and mistakes (your’s and other’s).

Instinct is recognizing pitfalls before they manifest into problems, recognizing winning solutions without having to explore and test endless options. It’s seeing balance, observing inconsistencies, and honing your design eye. It’s having good aesthetic taste, but knowing how to adapt your style on a whim.

Design instinct is the sum of all the tools you need to make great design decisions in the absence of meaningful data.

Clicks and conversions aren’t your only goals

Not everything that can be counted counts. Not everything that counts can be counted.

Data is good at measuring things that are easy to measure. Some goals are less tangible, but that doesn’t make them less important.

While you’re chasing a 2% increase in conversion rate you may be suffering a 10% decrease in brand trustworthiness. You’ve optimized for something that’s objectively measured, at the cost of goals that aren’t so easily codified.

This point is perfectly illustrated by a story by Braden Kowitz, a design partner at Google Ventures (via Wired):

One of my first projects at Google was to design the “Google Checkout” button. With each wave of design feedback I was asked to make the button bolder, larger, more eye catching, and even “clicky” (whatever that means). The proposed design slowly became more garish and eventually, downright ugly.

To make a point, a colleague of mine stepped in with an unexpected move: He designed the most attention-grabbing button he could possibly muster: flames shooting out the side, a massive chiseled 3-D bevel, an all-caps label (“FREE iPOD”) with a minuscule “Checkout for a chance to win”.

This move reset the entire conversation. It became clear to the team in that moment that we cared about more than just clicks. We had other goals for this design: It needed to set expectations about what happens next, it needed to communicate quality, and we wanted it to build familiarity and trust in our brand.

We could have easily measured how many customers clicked one button versus another, and used that data to pick an optimal button. But that approach would have ignored the big picture and other important goals.

It’s easy to make data-driven design decisions, but relying on data alone ignores that some goals are difficult to measure. Data is very useful for incremental, tactical changes, but only if it’s checked and balanced by our instincts and common sense.

When data-driven design gets fugly

Ever used Booking.com?

Search for a hotel and you’ll see every listing plastered with a handful of conversion triggers and manufactured urgency/scarcity indicators. Amongst all that crap, it’s difficult to find the real info you’re looking for. It’s a terrible user experience for me. I’m sure many others feel the same.

But they must have reliable data that says it works. Conversion rates must go up with each chintzy trigger they cram in. Data says: add more urgency messages, add more upsells, more, more, more. User experience says: less, less, less, just show me what I’m looking for.

What’s going on here?

Data has become an authoritarian who has fired the other advisors who may have tempered his ill will. A designer’s instinct would ask, “Do people actually enjoy using this?” or “How do these tactics reflect on our reputation and brand?”

Booking.com’s brand is cheap deals, so they’re not worried about cheap tactics. If those tacky labels stoke enough FOMO to get a few more bookings, they’ve won. It doesn’t critically damage other goals if they’re perceived as a little gaudy in the process.

But not every business has the luxury of caring only about clicks and conversions. You may need to convey quality and trust. Or exclusivity and class. Does cramming in data-driven conversion triggers serve those goals too? Or would building a more focused and delightful user experience better speak to your user’s needs?

Data-driven sameness

Digital interface design is going through a bland period of sameness. I see it in my own work, and I worry it’s becoming hard to escape from.

You could blame Apple and Google for publishing good design systems, and then everyone else trying to look the same. You could blame WordPress for the proliferation of content-agnostic templates — pulling apart the age-old marriage of content and designer. Or you could blame platforms like Dribbble that amplify trends and superficial eye-candy.

I’d argue that data-driven design also plays a role in why all websites look the same.

We’re all scared to experiment and reinvent the wheel, because data’s already proven that the wheels we’ve got work well enough. When our Agile processes are geared toward efficiency, it’s too costly to prototype and test innovative solutions. So we blindly churn out the same tried and true solutions over and over again.

Design “process” has replaced instinct as the new skill to fetishize. Some say that everyone is a designer if they can only follow the same processes we do. While that’s not true, it still leads to design decisions being made without the temperance of a professional designer’s instincts and experience. It creates more generic-looking interfaces that may perform well in numbers but fall short of appealing to our senses.

We’re all scared to experiment and reinvent the wheel, because data’s already proven that the wheels we’ve got work well enough.

Data is only as good as the questions you ask

What makes data so dangerous is that your input grossly colors your output. If you ask the wrong questions at the wrong time, or to the wrong people, you draw bad conclusions.

First adapters and eager user-testers don’t necessarily behave the same as your average user, so even when you are asking the right questions you can get tainted data.

The most empathetic designers — who are convinced they see the product the same way as their users — don’t behave the same. They know the product too intimately. They can’t see it objectively anymore. They can’t become naive.

Beware of misleading data. It’s only one source of info, and it’s only ever as good as your collection methods. Rather than blindly following the conclusions of big data, back them up with other sources (or at least common sense) before charging ahead with your shiny validation in hand.