The Technologist’s Dilemma

In 1997, Clay Christensen published the landmark book The Innovator’s Dilemma. In this book he went through a variety of industries (hard drives, backhoes, steel mills) and demonstrated the frustratingly similar pattern of how companies innovated, then stagnated – for entirely rational reasons. The key graph looks like this:

The idea is that for any product, there’s a bare minimum feature set required for it to be considered part of a category. So, for word processors, you need paragraph formatting, pagination, printing, etc. But after a certain point, no one cares about new features – the Microsoft Word team can keep coming out with new versions, but all they’ve done for the past decade is piss people off with unwanted design changes. Minimum requirements increase as a category matures, but not as quickly as product teams can pump out features.

In the above graph, Product A is the clear market leader. It has the first mover advantage, and is way ahead of Product B in terms of feature set. In fact, it’s so far ahead, and so feature complete, that there’s no way B can compete with it – in the beginning, B doesn’t even qualify as a competitor. So, in order to survive, B has to find other markets that don’t have the same requirements. If we’re talking about word processors, maybe it will target the text editing, or screenplay writing market, or something. Over time, if B survives, it will continue to add features, and will eventually hit that bottom line. At which point several things will be true: 1) B is generating revenue from a market in which A doesn’t compete; 2) over time, B will continue to make meaningful improvements (because it’s between the min and max functionality lines); 3) A is no longer making meaningful improvements (because it’s above the max line); 4) B is going to start taking business away from A. This spells big trouble for A.

The problem is that all the incentives are for companies to keep working on existing, successful products, even after they’ve passed the upper line. It’s easy to justify continuing work on the main breadwinner, people want to be associated with the main product, and even if you’re able to start a skunkworks project, it’s unlikely to impact revenue meaningfully for a long time – which makes all its best people high value targets to be poached for “higher priority projects”. There are many, many more reasons, and you can read the book to get the details, if you’re so inclined.

But that isn’t what I wanted to talk about.

In my last post, I discussed the various stages of learning. I’ve been thinking a lot about this recently, as I’ve been going through an accelerated learning process for a little over a year now – first by moving into DevOps at TripAdvisor, and now with a change to a startup with a radically different stack. And it occurred to me that the above graph was actually a pretty good model for the experience.

When you’re just starting out, you’ve got nothing to lose – you suck at everything, and if you’re going to be successful at all, then you just need to choose something and get started. Over time, you make it past the tutorials, basic projects, naive implementation decisions, poor architecture, and obvious anti-patterns, and achieve a reasonable level of professional competence. Congratulations! You’re suddenly aware of how much there is to learn, but when you’re in the green zone, you’re using your skills in a meaningful way. Choosing a diverse set of projects, continuing to read up on the technology, learning from more senior engineers, etc., will help you to continue to develop your skills. Until at some point – it isn’t a solid line, of course – you achieve mastery of your subject.

Continuing to develop past this point will provide asymptotically diminishing value. Yes, you can master the finer points of obscure libraries that you will almost certainly never use, or data structures that – while fascinating – are impractical for real-world use, but at some point these aren’t actually making you a stronger programmer. For that, you need to explore other paradigms.

Unfortunately, the graph above shows why this is hard. All of your short-term incentives are to continue to work on skill A – this is where you’re most effective, where you’re most confident, and have the greatest sense of self-efficacy. You feel as though you’re earning your paycheck, not to mention the respect of your coworkers. Switching to skill B is going to send you back to the bottom of the heap – of course, your existing knowledge should help you learn it faster, but you’re going to have to spend a fair amount of time being both the most expensive, and least capable, member of the team.

Switching classes

In the back of the old Player’s Handbook (1st edition, baby!), there was a special rule that allowed your character to switch to a new class. The idea was that after you’d become an Nth level fighter, cleric, magic user, etc., you could go back to being a 1st level something else, and start building up from scratch. You’d have all your old abilities, but the catch was that using any of them would rob you of all experience points for the adventure – you got to keep all your hit points, but otherwise had to come by knowledge the hard way. Theoretically, once you’d matched your old level in your new class, you could start using both. I never saw anyone use this rule.

Learning languages

Language learning is hard. There’s the grammar, vocabulary, listening comprehension, in some cases an alien writing system… But in addition to all that, there’s the fact that you’re going to be treated like (and feel like) a child until you can get to competence. This is extremely tough on the ego, and unless you’re able to switch off the part of yourself that cares, you’re not going to make it.

Change is hard

Switching to DevOps a year ago was hard. I was a senior hands-off engineering manager moving into a player/coach role on a team using technologies I’d never even heard of. I spent hours sitting in the server room installing hard drives, setting up RAID, changing controller batteries, and so on (somewhere in the back of my mind I kept hearing, “show me ‘wash the car'”). Every member of the team, all of them far more junior, knew more than me in their area of specialty, and it was hard not to feel that I wasn’t earning my keep. But I learned.

Joining a new company as VP of engineering has been similarly hard. I’m the most senior person on the team, I’m supposed to be providing technical leadership, but it’s an unfamiliar code base, a new stack, and my “relevant” Javascript knowledge is ten years out of date (thanks, TripAdvisor, for choosing Mootools as your Javascript library <smh>). When joining a new team it’s important to gain the respect of your peers, reports, and superiors, which is hard when everyone else – including people with almost twenty years less experience – knows more than you do. But I’m learning.

This is absolutely not an attempt to convince you of how brave, noble, or wicked smaht I am. Nor is it a cry for help from a big company manager thrown into the deep end of a hands-on role at a twelve person startup. Rather, it’s a message of hope. Being at the bottom of the graph doesn’t always feel great, but you have to remember that you will get through it. Yes, you’ll have to grind away at it for a while. Yes, you’ll have feelings of inadequacy. Yes, you’ll live with the constant fear of failure. But this will all pass. It’ll be hard, but a week, a month, a year later, you’re going to be a different, more interesting, more experienced person than you would have been if you’d stayed safely above the upper line.