Limbo: Scaling Software Collaboration

Limbo lower now

Limbo lower now

How low can you go?

I sit here in CodeNode London, a fitting place to think about collaboration and technology. London is muggy and gray. People are bustling from importance to importance, clad lightly as they try to escape pursuing perspiration.

The story I tell today, the vision, has much to do with collaboration and with perspiration. Usually my stories feature the unlikely interaction of two ideas. Today’s story combines something like four or five. You might want to pour yourself a glass of something cold.

tl;dr To scale development, make the changes which propagate between developers (and from developer to production) as small as possible. Hence the name Limbo, where the question is, “How low can you go?” (Thanks to Saurav Mohapatra for the name.) (We experimented withLimbo on the Cheap tl;dr it’s fun.)

Idea 1: Scale

During my tenure at Facebook, 2011 to February of this year, I watched the engineering corps swell from 700 souls to more than 5000, with no end to the growth in sight. Me being me, I started to think about engineering in seven more years, or 40,000. Round up and the question becomes, “How do 100,000 engineers work on the same system?”

Facebook’s software engineering workflow is fairly conventional:

1. Create a diff.
2. Get it reviewed and approved.
3. Put it in production.
4. Monitor for unexpected consequences.

As with many code-review/approval workflows, the review step introduces unpredictable delay, leading to:

• Larger diffs
• Stacked diffs
• Multi-tasking

Each of which leads to waste.

Idea 2: Smalltalk

As my tenure at Facebook was coming to an end, I wanted to leave the gang with something to think about. Alex Kotliarskyi convinced me to give two presentations on Smalltalk. Smalltalk occupies a different point in design space than most software systems widely used today. “ Computer support for the creative spirit in everyone ”? What’s that about?

To make sure I was spiritually prepared to present the essence of Smalltalk, I undertook to reimplement Smalltalk from scratch, at least as much of it as I could get done in two weeks.

I got a fair ways along on the project, including a syntax tree to bytecode compiler and a bytecode interpreter. What was most surprising to me was that I began to have visions as I coded. I had begun the project from a purely personal perspective, but it turned out to be intimately connected to the questions of scale rattling around in the back of my brain.

Idea 3: Conway’s Game of Life

Changes propagating

I’m going to assume you’re familiar with Conway’s Game of Life. (If you didn’t waste ridiculous amounts of your youth on Life like I did, are you in for a treat now.) As I was coding, I kept having the vision of Life, with changes propagating across the surface.

I didn’t know what this vision meant at first. Then I thought, “What if each pixel was a programmer? What if the changing state of a pixel were a change to the program? You could have lots of programmers working at the same time.”

Idea 4: Prune, or Tree Transformations

What are these changes that ripple from one programmer to another? I was looking for finer grained changes than today’s typical diffs. I also wanted less chance of editing collisions than offered by textual diffs.

A few years ago Thiago Hirai and I worked on a code editor that wasn’t a text editor . All editing operations were transformations of the syntax tree. Surprisingly, about 100 operations sufficed to cover all common edits. What if the unit of change was the tree transformation? One or at most a few transforms should be enough to move the program from one “correct” state to another. Transforms compose nicely. Genuine conflicts are easy to spot (like renaming the same variable to different names).

Idea 5: Guilds

The highest leverage application of Limbo is Limbo itself. Accelerating the development of the system for accelerating development compounds twice. However, this inevitably leads to tricky migration situations. If you are changing the resources that allow change, you might have a bad time.

A possible social solution to such trickiness suggested itself to me while watching my oldest daughter and her partner play World of Warcraft. WoW has developed an elaborate system of guilds. Guilds plan raids, execute raids, and perform after action analysis of raids. I would expect Limbo to spawn similar social structures — the Garbage Collector guild, the Performance guild, and so on.

Limbo

Limbo is live, shared programming. It balances precariously between two principles:

• Everyone is working on (and production is executing) the same program, represented by a single abstract syntax tree.
• No one is allowed to cause others (including users) problems.

Both principles are ideals to be successively approached.

The changes that propagate through Limbo are transformations of the abstract syntax tree. This encourages a “make the change easy, then make the easy change” sequence of changes.

The prohibition against causing other people problems implies quick and reliable local testing of changes (for those changes that might cause problems).

Limbo is an exercise in learning, hence the dogmatic stance that everyone is working on the same program all the way out to production. Imposing the harshest constraints on one dimension will encourage the greatest innovation on another. Even if Limbo turns out to be impractical, attempting to achieve it is certain to provide valuable spin-offs.

Open Questions

• Programming language. Can Limbo be applied to a current language or does it need its own language? Does Limbo work better with object languages or functional languages?
• Tools. Editing Limbo in Limbo will be significantly harder than using existing editing tools, but will cause the most rapid improvement in Limbo-based editing. Will Limbo get better enough fast enough?
• Inclusion. Can Limbo’s social structures evolve to accept contributions from a wide variety of people? Large portions of today’s engineering talent is actively discouraged in current social structures.
• Topography. What is the best connection strategy for propagating changes? (I’m assuming the Game of Life’s 2D toroid is not it, but who knows?)
• Business model. Lack of a way to engage capital as well as talent crippled the growth of Extreme Programming. How does Limbo attract capital?
• Architecture. Short-term, Limbo needs ways of isolating breaking changes — sorry, that change will break production. How does Limbo encourage strategic feedback — sorry, that’s the wrong way to approach the problem? I’m imagining something like Facebook’s News Feed, but for program changes.

Next Steps

You can begin experimenting with Limbo today.

• Make micro-commits. Delete an irrelevant comment? Commit. See how long it takes that change to go through and then optimize.
• Sort commits into “couldn’t possibly cause problems” (you’ll be mostly right) and “might cause problems”. Treat them differently. Rearrange your workflow so you have mostly “couldn’t possibly cause problems” commits.
• Think of code editing in terms of tree transformations instead of manipulating a 2D grid of characters.
• Make hard changes easy instead of just making hard changes.
• Start building your own Limbo. Use a language or invent one. Use a VM or invent one. Use a phone. A tablet. A laptop. A remote server.
• Experiment with topologies for change propagation.

The idea that we as developers have already found the best of all possible social structures for collaboration seems absurd to me. Only if we explore alternatives to “blocking review of textual diffs” do we give ourselves the chance to find something better.

How low can you go?