Summary.    The lean startup approach for developing new products or businesses has become popular among companies large and small. But is the approach really effective? An extensive study found that it is — but the approach doesn’t work equally well for everyone. While...

The lean startup approach has taken the entrepreneurship world by storm. Even established companies such as Dropbox, Slack, and General Electric have embraced it as a way to rapidly test and refine ideas for products or businesses. But is lean startup all it’s cracked up to be?

In a recent Strategic Entrepreneurship Journal article, we published the first study to experimentally test the key components of lean startup. Using data from the National Science Foundation’s I-Corps program, which uses the lean startup method to help academic scientists and engineers commercialize technologies they’ve developed, we found the approach largely works as promised: It prompts teams to explore hypotheses and converge on ideas. Further, teams who engaged with the method more intensely had more operational success (that is, established a company and employed more people) in the 18-month period following the I-Corps program.

But there is a catch: The approach doesn’t work equally well for everyone. While educationally diverse teams — those whose members have different kinds of degrees (engineering, medicine, and so forth) — embraced its experimental, learning-by-doing nature, adding people with business degrees to the mix had both positive and negative effects. MBAs tended to resist the method at first, we found, likely because their training emphasizes learning by thinking. Yet teams with MBA members who started using the components of the method (particularly customer interviews) were more likely (relative to teams without MBA members) to improve the teams’ business ideas. This suggests unexpected benefits if we get MBAs to embrace the method in the first place. Based on our research, we also developed evidence-based recommendations for how innovators can overcome the method’s limitations and make the most of it.

First, a quick primer: The lean approach was devised in the early 2000s. Its key features are formulating hypotheses about different aspects of the business and “getting out of the building” to probe each hypothesis by interviewing potential clients and consumers and quickly pushing testable prototypes to them. Using this method, companies can avoid devoting time and energy to ideas that won’t work in the real world, and instead emerge with a viable plan of attack.

Putting Lean Startup to the Test

We performed our analyses using behind-the-scenes data from 381 I-Corps participants representing 152 teams from 16 different cohorts of the program. This included biographical and demographic data, such as team members’ educational backgrounds. We also tracked the hypotheses, interviews, and decisions about business ideas that each of the teams made weekly. In addition, we followed one cohort in more depth, conducting weekly surveys of all teams. We also investigated how teams fared after leaving the program.

These data allowed us to measure three key dimensions of the lean approach each week: hypothesis formulation (the number of new assumptions teams developed about their business model), hypothesis probing (the number of interviews conducted about those assumptions), and idea convergence (the number of items teams either added to their business model because they had been proved or removed because they had been disproved).

Our data show that the lean startup method does what it promises to do. We found a positive correlation between teams’ interviews of potential customers in a given week and their convergence on a particular idea the next. In other words, the more interviews the teams conducted, the faster they settled on whether a business idea was worth pursuing — something advocates of the approach assume but which has not been empirically proven until now. Simply put, it pays off to leave the building and ask customers what they think of your idea.

What’s more, talking to customers appeared to create a useful feedback loop: While conducting interviews, teams got brand-new ideas that became new hypotheses. This may come as a surprise to innovators who’ve long been told that customers never provide brand new ideas. It turns out that they can and do.

One common criticism of lean startup is that the feedback loop never stops. As an innovator, you may exhaust your time and resources on testing hypotheses rather than building a business. Our data did not support this critique: We found a negative correlation between teams’ idea convergence one week and hypothesis formulation the next. In other words, there is a natural stopping mechanism baked into the method: Teams gradually settled their open questions about their business ideas.

Why Team Makeup Matters

But all of these useful downstream effects of lean startup rest on getting teams to propose and probe hypotheses in the first place. Here we saw the fates of teams with different educational backgrounds diverge most sharply. Teams with at least one member who held an MBA degree struggled relative to others and, as a result, formulated fewer hypotheses and converged more slowly on business ideas.

Why do MBA teams tend to struggle? While our study did not directly investigate the causes of MBAs’ struggles, our interview data suggest possible reasons. Traditional business school curriculum emphasizes learning by thinking — for instance, analyzing case studies — rather than learning by doing; as a result, MBA teams may believe that by thinking through a business model, they’ve already validated it, so no hypotheses are required. We also suspect that MBAs may feel reluctant to hypothesize and probe because they view themselves as business experts who know more than customers do.

But it’s not all bad news for MBAs: Interestingly, when they did engage with the lean startup process, their teams benefited. In fact, probes by such teams were more likely to result in new ideas and business idea convergence than probes by teams with no MBA members, which suggests that MBAs are especially good at interpreting the results of customer interviews. MBA teams were also more likely to make major changes to business models in response to probing. In other words, while MBAs may have a hard time adopting lean startup at first, they can learn a lot from it.

How to Make the Most of Lean Startup

So what does this all mean for aspiring innovators considering the lean approach? Here are three ways to make the most of it:

When it comes to developing hypotheses, focus on quality, not quantity. One of our more surprising findings was that teams who developed lots of hypotheses tended to investigate fewer of them. (Educationally diverse teams were the exception to this pattern.) Although our data did not directly reveal why, we suspect it is because it’s much easier to probe a few clear, thoughtful hypotheses than lots of vague, slapdash ones. To yield all the benefits of probing, lean teams should focus on developing a smaller number of hypotheses they really want to test.

If you’re an MBA, prepare to eat some humble pie. MBAs and lean startup aren’t necessarily a natural fit. But MBAs can find great success with this approach. So, if you’re a business school alum, prepare for some discomfort at first — but know your skills will give you a boost when it comes time to analyze the results of your probe.

Embrace diversity. We were struck by the unusual success of educationally diverse teams. Homogenous teams (for instance, those made up of only engineers or only MBAs) may be overly optimistic about their initial ideas and less willing to probe them. That’s likely because a group of people with similar backgrounds may be prone to groupthink. But educationally diverse teams don’t have this problem. Thanks to members’ varied backgrounds, they are less susceptible to groupthink and have less attachment to initial beliefs, likely making them especially willing to embrace the experimental nature of the method and probe their ideas. While our data were mostly focused on educational diversity, we believe other types of diversity would yield similarly positive consensus-busting effects and encourage teams to question what they think they know.

So if your innovation teams are made up mostly of people with similar backgrounds, you should broaden them. And if you’ve got an MBA on your team, know that it may take a little convincing to get them to develop and probe hypotheses — but that you’ll all benefit in the end.