Start with data: how to incorporate data into your design process

6 tips to help you become more data informed and foster better working relationship with data scientists

At Coursera, I had the opportunity to work with the team to improve the learning experience. Throughout the process, I’ve learned how to better collaborate with the data science team and became more fluent in quantitative data. I want to share some key takeaways from working closely with data scientists, as I believe that incorporating data as part of the design process is important to help guide product decisions and impact.

Don’t worry if you’re not good at numbers, you don’t have to be. A better working relationship means a better understanding of what data science does, and how you can use data in your design process. Once you become more familiar, you will know what questions to ask, and when is the right time to involve your data team.

This doesn’t mean that data is all you need. You still need user research and qualitative data to understand what are the motivations behind certain behaviors. Data is simply there to uncover users’ patterns and enable you to make an informed design decision.

Aaron Gitlin published a very detailed article on how to become a data-aware designer, and how to use data in every step of your process. Another good article to read is how to design using data, Joanna provided some good tips when working with data. To learn more, here is an in-depth book on Designing with Data by Rochelle King, Elizabeth F Churchill, and Caitlin Tan.

In this article, I’m going to share tips on how to become more data aware in your design process, and what you can do to foster a better relationship with the data team, and how to use data .

Get familiar with data and terminology

As designers, we need to balance between user needs and the company business goals. Every project is probably tied to specific company metrics. It’s important to understand what the metrics mean so you can use them to effectively communicate with others.

For example, what does user activation mean? How do we calculate active users? What is the churn rate?

When you understand the meaning behind this number, you become empowered to hold an effective conversation with your data team because you’re speaking the same language. The other added benefit is that you’re more likely to satisfy the business goal while focusing on addressing user problems.

Ask the right question

When you’re given a project, instead of jumping to a solution, you should make sure you understand the problem space by asking questions. The product manager might paint you a picture of the problem and vision, but don’t stop there, dig deeper. Ask your data team about existing data analysis to better understand the pattern. Often the data team might already have a dashboard or notebook from past analysis and experiments. Start by reaching out with clear questions and what you’re trying to learn.

For example, you can ask “What’s the path to this particular feature?” “Where do people drop off in the funnel?” “What are the common tasks users do on this page” “Where are people spending the most time?”

If your project team doesn’t have any tracking data, then it’s time to have a conversation with the team, and start strategizing how and when to use tools to get more data. These data will become important when making your design decisions. Keep in mind that having data is not enough, you would still want to look at qualitative data such as customers support tickets and talk to relevant stakeholders to see the full picture.

Have clear hypotheses

Once you have alignment on the problem you’re solving, it’s important to have a clear hypothesis, and how you would measure success.

1. Hypothesis: We believe that [creating this experience], for [persona], we will achieve [outcome]
2. Result: how do we measure results? What’s the evidence to declare if the hypothesis is valid or invalid.

For example, the team could have a hypothesis like — We believe that [letting users set up their learning goal] for [people who’re looking to switch their career], we will [make them feel more motivated and committed, therefore become more active in their learning.]

And how we’re going to measure the result — for example, the team designed an A/B experiment, and let’s say the result we’re looking for is a 5% increase in average learning sessions.

Having a clear hypothesis 1) allows the team to have clear alignment on the goals we’re trying to achieve 2) can be tested, measured, and validated or invalidated.

Involve your data team every step of your process

There are a lot of benefits when involving cross-functional teams in the design process — different perspectives, getting buy-in, creative ideas. Likewise, you’ll benefit from including a data expert in your design process at every stage.

1. Design sprint — data team can give you insights into users’ patterns and behavior, like where are people dropping of in the funnel, what types of users are most active during a certain time of the day. These insights can help uncover or prioritize areas that need to be addressed.
2. Problem scoping and hypothesis — data scientists can usually recommend metrics that are measurable. You might want to look at multiple metrics instead of a single one to make sure to head in the right direction.
3. Brainstorming workshop — because they’re familiar with the data and user behavior patterns, often time they have great ideas.
4. Design (user flow & interaction design) — there might be design decision that’s directly impacted by what you want to learn, or by data modal. For example, if you’re designing a recommendation system, your data team might want a feedback loop to enhance the modal. Therefore you might need to design a way to collect user feedback.
5. Analysis & iteration — to further improve and iterate your product, you need to have more insights on the previous iteration. What’s the adoption rate? Did it reduce drop off? By how much? Where are people clicking the most? What do people do next?

List out all your questions to get the learnings you want

This will help you start a conversation with your data team. Some of your questions might get answered from the experiment, some might need you to tweak your design in order to get those learnings. Get feedback from data scientists to know what’s feasible, what’s out-of-reach, and what might become available with your new design. It’s also a great way to gather inputs from other cross-functional team members. For example, if I’m designing for a learning goal selector, these are the questions I might ask:

1. What is the most popular option? Among what group of users?
2. Which option is most effective?
3. What is the completion rate for each option?
4. What’s the percentage of users who opt-in for the experience?
5. Within the opt-in users, are there any differences between the different course domains?

It takes me some times to become more familiar with our data and practice to ask better questions.

Review analysis

After a feature has been implemented, usually your data team will provide you with A/B testing results, or other types of quantitative results (e.g., satisfaction rate, open rate, click-through rate, etc.) Regardless of whether the outcome is positive or negative, it’s important to spend time understanding what the data is telling us.

• What do we learn if we fail? How do we apply our learnings in future projects?
• What do we learn if we succeed? What’s the next step?

Don’t be afraid when the result came back negative, you will still learn a great deal from the experiment. Gather the whole team to discuss and understand the result.

• Is our hypothesis correct? Do we need to change our hypothesis?
• Look at the implementation details? Is there any friction in the user experience?
• What are some of the assumptions we made about our users and their needs?