00:00

01:06:21

Matt Provost is the author of the Leanpub book Rust From the Ground Up: Real World CLI Programming in Rust. In this interview, Leanpub co-founder Len Epp talks with Matt about his background, how he started his career in tech, moving to new countries, working for Weta Digital on the tech side for the production of Avatar and other movies, his book, and at the end, they talk a little bit about his experience as a self-published author.

This interview was recorded on January 11, 2022.

The full audio for the interview is here: https://s3.amazonaws.com/leanpub_podcasts/FM195-Matt-Provost-2022-01-11.mp3. You can subscribe to the Frontmatter podcast in iTunes here https://itunes.apple.com/ca/podcast/leanpub-podcast/id517117137 or add the podcast URL directly here: https://itunes.apple.com/ca/podcast/leanpub-podcast/id517117137.

This interview has been edited for conciseness and clarity.

Len: Hi I'm Len Epp from Leanpub, and in this episode of the Frontmatter podcast I'll be interviewing Matt Provost.

Based in London, Matt is an IT professional and public speaker with over two decades of experience working for a number of companies around the world, inlcuding for Weta Digital, Yelp, and Lyft, and he currently works for Woven Planet, a self-driving technology company.

You can follow him on Twitter @hypersupermeta and check out his website at rftgu.rs.

Matt is the author of the in-progress book Rust From the Ground Up: Real World CLI Programming in Rust. In the book, Matt uses a practical approach to introduce readers to the Rust programming language, which has a notoriously steep learning curve, showing readers how to build powerful applications and develop intuition and confidence in Rust programming.

In this interview, we’re going to talk about Matt's background and career, professional interests, his book, and at the end we'll talk about his experience writing and self-publishing.

So, thank you Matt for being on the Leanpub Frontmatter Podcast.

Matt: Thanks, it's great to be here.

Len: I always like to start these interviews by asking people for their origin story. So, I was wondering if you could talk a little bit about where you grew up, and how you found your way into your career in technology?

Matt: Yeah, happy to. I grew up in Connecticut in New England. And of course - I've kind of come full circle and live in Old England now, in London, with a lot of stops along the way.

I went to Indiana University for my undergrad, and majored in History and Art History. So, nothing to do with computers.

But I grew up surrounded by computers. My dad was sort of one of the first crop of Computer Science people going through a program in the 70s. And so we grew up with like the TRS-80 and a color computer - and then graduated to IBM XT clone, and all that kind of stuff. We always had computers in the house, ever since I was a little kid. My dad was actually publishing computer games when I was growing up, for the color computer.

Len: Oh wow.

Matt: And that, he used to do in his spare time. So, yeah - I was kind of surrounded by it. I tried to get away from it a little bit in college, but it dragged me back in.

I ended up in Los Angeles for the first dot com boom in 2000. I think I actually moved to LA the week of the stock market crash, for the first crash there in 2000. But I made my way around, and had a series of jobs in system administration, doing a lot of Solaris and all that kind of stuff, some networking - things like that.

Eventually in 2005, I immigrated to New Zealand, without much of a plan. I moved to Christchurch, where I met my wife. And then I ended up, through some contacts, getting to know some people at Weta Digital, which is Peter Jackson's VFX studio. So, the people that did Lord Of The Rings, and all of that.

Finding good tech people in a small country like New Zealand is a challenge. So they were always looking out for people with any kind of experience. I talked to them, and they had some top secret project that they couldn't really tell me about, but that they needed help on.

I moved to Wellington to do that. That turned out to be Avatar. I spent the next couple of years working behind the scenes on all the infrastructure to render Avatar, which was a ton of work, in 3D and all of that. We won the Oscar for that. That was pretty fun.

I stuck around in that industry all the way through the first two Hobbit movies, and made a bunch of movies in the middle. I did the first two Hobbit movies, and then got really burnt out before the third one. I said, "I don't have it in me to do one more."

We decided to see the world, and do a bit of travel and stuff. I moved to London with one of the companies that we'd been working with in the effects infrastructure space. I worked for them for a couple of years, before going to Yelp, where I decided to get back into engineering management. Take a little bit less of a technical role, and more of a people role.

Then I moved over to Lyft. Part of their Level 5 self-driving division was in London, and they needed an engineering manager to run the London office. How could I say no to getting involved in self-driving? I took that.

Then, I guess, a little less than a year ago, we were acquired by Woven Planet,which is a division of Toyota, which is the world's largest car company, so that's what we're working on now.

I always tell people - if I try to plan out my career, I mean there's just no way that it would all fit together. It's just one of those things. One thing leads to another, and a conversation with someone leads to another conversation, and - it's organically grown in a really interesting direction.

Len: Thanks very much for sharing that really great, very brief synopsis of so many different things and so many different paths you've gone down. One thing I wanted - to start at the beginning, I guess? You were studying Art and Art History in Indiana, I believe - just looking at LinkedIn here. You've made your way to LA. How did you get a job like in IT, without having a formal background in it at the time? As you mentioned, it was a specific time. There was a big dot com boom going on.

Matt: That's exactly it. I just timed it right. I was all hands on deck. I had a real background in tech. I was working in the computer lab at school, as a part-time job. They would have people around the lab. I've been doing that throughout my studies. I kept a bit of interest there.

I think, right towards the end of my college career, I realized that I didn't want to go to grad school for Archeology or something like that. There's just no career there.

I think my last year of school - I did some intro to Computer Science courses, and picked up enough of the basics. But yeah - I managed to tuck myself into a smaller job, I think - looking for anybody that knew anything about anything, to get involved at that point in time. So I just timed it right.

Len: It's so interesting when you say, "They." "They were looking." That includes - Warner Bros. was looking for people. Is that right? There's this really fascinating detail on your profile, where you were sitting on top of hundreds of Warner Bros. websites at a certain point?

Matt: Yeah, that's right.

Len: I think this might be one of the podcasts where people will know exactly what I'm talking about, but you were actually sitting on top of the Space Jam website, which is a part of web history.

Matt: Yeah, that's right. I leave that on my LinkedIn as like, an Easter egg, to see if any interviewers ever ask me about it. Almost nobody does. Well done for finding it.

But yeah, I ended up as one of the webmasters at Warner Bros., which, at the time, was part of AOL. It was all running on these big iron Solaris machines, which is what I specialized in. It's hard to imagine now, but there was a time when Linux was really a toy. The first dot com boom was all pretty much built on Sun Solaris. That's what I was specialized in at the time.

And, yeah - I ended up going to Warner Bros. Every time a movie came out, they would hand us another website to run. And Space Jam was one of those.

But probably the most interesting one was Harry Potter. I was webmaster of the harrypotter.com website when the first movie came out. It was actually the busiest site on the internet for like a month, because every kid in the world was going to harrypotter.com. We had all kinds of games and different videos and stuff for the premiere. That was my introduction to scale, and trying to keep up with demand and everything else. Which really helped me later on when I was doing things like movies, which is all about scale.

Len: I was going to say, that's a super fascinating entryway into it. I mean this was, in a sense, compared to now - it's a very American metaphor, but it was very cowboy days, where if you were a guy who could run a website, there were people desperate to have you running their website.

Running a website back then meant something very different from running a website now. I was wondering if you could talk a little bit about what you did as a webmaster back then? Was your job to watch traffic and make sure there were enough servers available to meet the traffic, and things like that?

Matt: Yeah. That's exactly right. It was very hands-on back then. I mean, all the way from the hardware level, right? Like buying servers and taking them off pallets and racking them up, and plugging them in and building the network, and all of that. We weren't that specialized, so everybody just had to do everything. But anywhere - hand compiling an Apache server with the options that you needed and everything, and getting that to run. Then getting the content from everybody else. Putting it there.

A lot of the stuff we were doing back then was Java. Java was the hot new thing. There used to be a real divide. This was like long before DevOps. There really was like a wall in-between the developers and the system administrators, where like none of the system administrators knew anything about Java. They were just handed these files and told, "Here, run this on your servers." You were like, "Oh, okay."

I actually spent a lot of time tuning the JVM. I mean it's all taken care of now, from what I understand.

But back then, there was a lot of manual tuning. Watching the memory consumption. Watching the CPU consumption. Tweaking some parameter somewhere and seeing how things went. There were no guidebooks. I mean, I was working my first job in 2000 when Google came on the scene. Before that, it was like AltaVista, and things like that. There was definitely no Stack Overflow, open source things to look up. When something went wrong, you were really on your own trying to figure it out as best you could.

Len: Was it books that you would turn to? Books and colleagues, basically, that you would turn to, when you encountered a new challenge?

Matt: Yeah. I grew up having these big, fat O'Reilly books all around my desk, that you were frantically flipping through and studying on weekends, and all of that stuff. Then, yeah, through your network, you'd work with people that had been around longer than you and had had maybe seen something. There was a lot of tribal knowledge. Somebody had worked at some place and knew some secret way to squeeze some performance out of something. You had to build up your connections to find all that stuff out.

Len: You mentioned watching. That reminded me, around that time, around 2001 or so, I had a friend in London who got a sysadmin job. He quickly developed a thousand-yard stare, because he was on call all the time. At least, that's how I saw it. Something changed in him, it was very intense. At any moment, something could happen. He had one of the old pagers. He could get a ping on his pager - and off he had to go, to maybe deal with something really serious. Or maybe it was a false alarm or something like that. Was that the work that you were doing?

Matt: Yeah, exactly. At Warner Bros., we had those original first-generation Blackberry pagers, before they had phones. It was just like a pager with like - I forget. Three or five lines of text, and a little thumb reel. The classic Blackberry keyboard. that was pretty high-tech. I mean, for back then - to get, over the air, alerts and emails and everything else. That's how we watched the website. But, yeah, you were on call 24 hours a day.

I can definitely remember - it was for a different company, a web hosting company - but for our company Christmas party, we went to Universal Studios. We were on one of the rides, I think there was like a Jaws ride? Where you're in like a little boat that's on these little train tracks, it goes through. I got paged in the middle of the ride. I was sitting there with the CTO. It's like, "What do we do?" He just opened the door to the boat in the middle of the ride. All the lights came up and alarms went off everywhere. The ride completely stopped for everybody on it, for safety. We just ran out of the ride, down some like little corridor that they had for staff only. We ran away, and had to go deal with the website.

Len: Oh my goodness. That's intense. But good to see you weren't out on the middle of the water with- even a robotic shark, who knows what might happen.

Matt: Yeah.

Len: That's really intense. My next question is a version of a question that often gets asked on this podcast. Which is - if you knew when you were in college, what a career you were going to have - do you wish you'd studied Computer Science?

Matt: That's a really good question. I haven't thought too much about it. I've been pretty happy with the way things have gone. I wonder - if I had studied more Computer Science, I probably would've gone down more of like a programming route. I really enjoyed my career doing system administration. It tended to attract this different group of people, who hadn't like exactly studied Computer Science, but were interested in computers and had figured a whole bunch of stuff out.

I think if I had really studied Computer Science, I probably would've been a programmer. Which also would've been fine. I'm sure I would've done okay, but it would've been a really different experience. I really enjoyed my time doing system administration. But it's one of those things, of - you ask people, like, "How did you become a sysadmin?" Everybody's got some crazy story. Because there's no traditional route to do it. There's no course you can study at a university or anything. There's no traditional way of doing it. People fall into it, and I'm one of those. So, no regrets, I guess?

Len: There's a podcast interview I did a couple of years ago, with someone named Dave Kawula, who was from Saskatchewan, the province in Canada where I'm from. He and his teenage brother were in this town called Prince Albert. somehow their parents learned there was a contract out to basically hook up some computers for the like local city hall, or something like that. I'm getting it wrong, but it was something like that. They were basically the only two kids in town willing to hook things up. That's how they got their first contract. they ended in sysadmin style careers, to this day.

It's just fascinating to hear that - I've never heard before that that's actually like - the particular instance is unusual, but that it's unusual is not unusual, that someone's origin was like that.

Matt: Yeah.

Len: So, as you said, you bounced around a few places and you decided to move to New Zealand. You said, "Without much of a plan." What prompted that?

Matt: It's funny, I talk about this sometimes. The exact motivations are lost in my own history. I don't really remember. It wasn't very well thought out. Part of it was, I think there was a buzz in LA. This was in the time of the Lord Of The Rings movies and everything. I think a lot of the crew from LA that'd go down to do it, would come back. I think I ended up at some parties or something, where people were like, "I just spent a year in New Zealand, it was amazing".

It was one of these things where I thought, "Hey, I wonder if I could go to New Zealand?" I went online and applied for a visa, and got accepted. I was like, "Oh, right - okay, I guess I'm going to New Zealand." There was a lot of paperwork after that. It took probably about a year to get everything straight to move down. But the initial decision was over about the course of a week, I think. And then I didn't have a job. I just packed everything onto a plane and moved down and found something to do.

Len: I'm just curious - did you have something like a working holidaymaker style visa, or something like that?

Matt: I'm not sure if they still have it. It was maybe something like that. But it was basically a path to permanent residency.

They have a point system. You hear a lot about this like [the Australian system]. New Zealand has a similar thing. They have a skill shortage list. Which - when you look at it, is really funny. Because there are things - it's like doctors, dentists, civil engineers - all this stuff. But it's also dairy workers and plumbers and construction drivers. Different industries in New Zealand can apply to the government and say, "There isn't enough local market to hire from. Can we get some visas, to hire people from overseas?"

But it doesn't necessarily connect you directly with an employer. It's just in the industry. Technology, whatever - computer programming or system administration - I can't remember how specific it was. It may have just been IT in general, was definitely on the list.

Film is perpetually on the list. If you do anything related to movies, you can always get a visa to go down there.

But at the time I wasn't in that industry. I was just a sysadmin. I applied under that. I think the rules are - once you get there, you have to stay for two years. If you stay for two years, then you can apply for permanent residency. I did that. Then I think after six or seven years, you can apply for a citizenship, which I did as well.

Len: So you'd been there for a while, and then you got linked up with a meeting at Weta for a secret project, that, as you mentioned, turned out to be Avatar. Which, I think - people listening who don't know - Avatar was - I think - as I understand it, a project that deliberately set out on James Cameron's part to innovate on technology. That must've been really fascinating.

I was just wondering if you could talk a little bit about what that involved? I watched a talk that you gave - this would've been six or seven years ago, or something like that. But you talked about - basically sitting on top of the biggest collection of supercomputers in the southern hemisphere, or something like that - at one point. Maybe not for Avatar, but at Weta. Can you just talk a little bit about what your particular role was in developing that movie?

Matt: Yeah. Lots of good stories about that. The supercomputer definitely was for Avatar - we'll get there. I went down, and they were working on the movie. There were some early shots being worked on and stuff like that. But they were just beginning to ramp up the infrastructure.

The way it works in the movie industry often, is, you start off using the last movie's hardware. They had all these servers and everything that they'd used for King Kong, which had been Peter Jackson's previous movie, that had come out a couple of years ago. So they had all these servers sitting around, and they were doing that.

But they knew that Avatar was going to be this game-changing thing. They were in the process of building a purpose-built water-cooled data center on some industrial land near the airport. That was under construction when I started.

Then we just started ordering equipment. Every couple of months, more pallets would show up. I was the new guy on the team. I got stuck unboxing and racking up all these servers. I ended up racking up, I think almost every server that we bought for Avatar.

Eventually we opened up the data center, probably a year or so before the movie premiered, and commissioned that. I was involved in all the different parts of it. There was a systems team. It was a really small team, for the amount of work that got done. It was really incredible how productive everybody was. I think there were like seven or eight of us, or something like that?

Sometimes we would sit - it's like an open book in the movie industry, because the credits are there at the end of every movie.

You could go, and you'd watch like a Pixar movie or something. It would get to the systems team, and they would have like 30 people. We were like, "What are they--? Like why would they have 30 people? There are only eight of us." You can see what everybody else in the industry is doing. There were some people that were more into the networking side. Some people that were more into the Linux side. Some people that were more into the storage side, network storage was a huge, huge thing. Some people that were more into the HPC compute side.

So, I got to build that data center. I commissioned the network. I think I ended up falling more on the network and storage side eventually. I worked on that, built that. We ended up having basically the biggest supercomputer in the southern hemisphere, to do Avatar. Like twice a year, there's, it's some industry body - I forget. But they published like this Top 500 Supercomputer list, which companies and institutions are always trying to get onto, and everything else.

To get an official result, you have to run this benchmark called, "LINPACK." Where usually the vendor does it, like IBM or HP, or something. We were using HP computers at the time. But usually the vendor organizes this commissioning exercise. They'd go onto the campus or whatever, and they'd install this huge supercomputer - with usually some government grant, or something like that.

Then to commission it, they run this benchmark, which takes like a week to run. Then you get some number out of it. Then you submit that to this body, and then they sort it and put it - put you on the top 500 list. In fact, that number is often written into commercial contracts. You'll often sign a contract, like, "We want IBM to build us a supercomputer that can benchmark at this number," or something.

The problem is, we never had a week where we could run a benchmark. The computers were rendering Avatar 24/7. Even five minutes of downtime was a huge deal. We just never had time to do it.

Basically what we did was, as we could bring on maybe a rack of servers and run the benchmark - or HP would run the benchmark across that rack. Then we'd get that certified as a supercomputer. In the end, we had seven - if you look at the old list from that time, we had seven supercomputers on the Top 500 list. All in the top, around the 100 mark. But you'll just see, like, "101, 102, 103, 104," it's like all these Weta Digital supercomputers. I think if we had run - we estimated, I think - if we had run the benchmark on the entire data center all at once, it would've been top 25, so a pretty big deal.

Len: That's super interesting. I'm going to try and ask, if I can, interesting questions about it. One thing that's a really interesting detail in there, is that it's one data center, but it's multiple supercomputers - arbitrarily determined by which racks you ran the tests on?

Matt: Yeah, it was basically just as we would get them shipped in. The movie industry is also always trying to spend like the least amount of money that they can. They would buy you a certain amount of computers, and you would do some of the movie. Then everybody would be looking at the charts and the graphs and seeing how much work we had to do, and everything else. Everybody would sit there and say, "Well, we're not going to get it done with this. Can we have another five million dollars to go buy another rack of servers?" They said, "Alright, alright." So we'd buy another rack and commission it, and benchmark it, and then add it into the system. That might last a few months.

Then we'd all look at the graphs again, and say, "We're still not going to get it done." I mean it was - for some of these movies, especially for like the Hobbit movies, I can definitely remember - we were adding millions and millions of dollars of equipment in the last weeks of the movie. Running it hard for like two weeks. Then we turned it off for months, once the movie was finished rendering. But we wouldn't have finished the movie without it.

At some point, they were like, "Well, we have to spend a couple of million to buy more whatever cores, just to get the movie done - even though there's only a week left," so -

Len: That's just so interesting to me. I'm reminded of a friend who was working on a PhD, on engineering degree. His research got postponed, because he lost supercomputer time, that took months to get back.

So, when you're talking about running 24/7 - I imagine there must've been like - were you ever on the receiving end of like, I don't know, "The spaceship team needs time - but we've already allocated it to the living tree in the forest team, so screw you - you're later in the schedule," or something like that? Would that be the thing you'd have to confront, or would someone else make those decisions before they came down to you allocating things?

Matt: Yeah, luckily not. There was an entire team dedicated to that, called the data wranglers. That was basically their job. We were running 24/7, and we were on call 24/7. We were working ten-hour shifts during Avatar, but they had a 24/7 rotation. They had staff there all the time to keep the renders going. They had to balance all those things.

They'd have the calls from their producers saying, "Hey, we've got a trailer to get out, drop everything. We need this, this and this in the trailer. Get those done as a priority." They'd evict somebody else's stuff, and then get an angry phone call from somebody else. But thank God I didn't have to deal with - we had a whole team buffering us from the end users. They did a great job.

Len: It's really interesting. When I was doing research for this interview, I learned a little bit from a paper you wrote about how you're managing a large installation, a big data center, things like that. But it's also people, right? When the computer's got a problem, it goes, "Hey people, I need your help." Managing that process itself, can be very up to you, which is something I think that's like - for people on the outside like me, we think, "Oh, it must be really technical. It must be very specified," and stuff like that. But a lot of managing those systems, is managing teams of people, and trying to do things like prevent burnout, and things like that. Because the machine might burn out. But if the person you need to fix it is burnt out, that machine's going to stay burned out.

Matt: Yeah, that's exactly right. The company and the industry was going through a bit of a shift as well. Historically - in the beginning - the company was basically set up by Peter Jackson to make his movies. It really came into its own, of course, during Lord Of The Rings.

I was working with a lot of people that had been there during that time. You would talk to everybody. They would work on a movie for like a year, because the movies came out one, two, three. Then you'd have the premiere in December, and everybody would get to see the movie. You'd have this big celebration. Then everybody would go to the beach or something like that for months, and recover. Then they'd trickle back in the next year, and get ready and finish the next movie. There's a pace to it.

But the way it works these days - I think across the industry there's just so much money involved, that it's just this machine, and you're just cranking out movies 12 months out of the year. So, you never have that downtime. There's too much money and resource being wasted, if you're just letting all the computers and everything sit there.

You go find, like, "Oh, we can sign up to do 50 minutes of this Marvel movie," or something like that, in the gap in-between the big movies. So things like burnout become a real issue. Because you never have that downtime.

In the system side of things, we used to do things - like in-between the movies, you could tear down all the garbage that you had put in to keep things running. There was this virtual duct tape all over the place. Sometimes literal duct tape. Sometimes just horrible old hardware that we had to get out of storage and plug back in, just to keep things running, or just to get us over the last little bit. Then once the movie was out the door, you could kind of, "Okay, let's shut everything down. Let's recommission everything and do it properly and reboot, upgrade the OS."

A lot of things just got put on hold. Even if critical patches or something were coming out like a week before the movie was done, there is no way you were going to shut things down to do that. But over time, it became this operation of like - right after that movie was another movie. So you never had the time to go back and do things, and everything just piled up. So, yeah - managing burnout was a really big thing.

Actually, that highlights a little bit, my transition into management that happened around then.

On Avatar, I was working on the team and building networks and racking servers and all that stuff. After Avatar, the manager of the systems team left, I think from burnout. I got handed the role. I was thrown in the deep end - a little bit of management, without much mentorship or anything. I took that over.

But in the end, I think the manger spent like the next three years in Thailand or something, on a beach, doing yoga and stuff, to get his head straight. Which - maybe if I would've known how stressful it was going to be, I wouldn't have signed up. But at the time, I had no idea.

Len: We're not going to talk about this forever, but I mean, of course, we could. I wanted to actually dive into a very specific detail there. You had a system called, "Nagios," I think it was called, that you used?

Matt: Yeah, that's right.

Len: I think I mentioned already, but I'll link to it in the transcript for this interview, where you talk about this. It's really interesting to me, to hear about the dynamics of it.

For example, I gather one detail is that someone would be typically on call for a week at a time. That meant that you could be woken up in the middle of the night by - it probably wasn't a pager by this point, but something would go off, and you'd have to check and see. Then you still had to make a judgement call. Like, "Is this a false positive? Is this really serious? Is there something that's going to take care of this anyway? Did someone else also get notified about this? Because they set it so they would personally be alerted. Because it's the part of the system that they know best, and they want to be taking care of it."

But also, I gather that people were - if someone was being paid by the hour, if they got woken up by - if they got alerted, they got paid for an hour. They had these murky incentives to add alerts to the system.

I don't think you ever described it as "becoming a tangled mess," but it became a problem that you had to deal with. Things were a little overly arbitrary. I was wondering if you could talk a little bit about -? I think there's a paper that you published at a conference or something like that. But about how you finally jumped on the problem, and tried to tackle it.

Matt: Yeah, it's funny that you found that old paper, good. Nagios was the industry standard of open source, the monitoring system. You would put all these rules in and say, "If you see this happen if this system goes offline, page this person." We said, "Take it in turns to do that." But it was pretty brutal. Especially when things were really busy. You would get woken up every night - while you were on call. I think with any of those systems, the tool wasn't the problem. It was the system.

But over time, you're just constantly adding, like, "Well, we never thought it could fail in that way before," but it does. Then you add an alert for it. The number of alerts just keeps increasing. Because you just keep catching more and more things. So more and more things are just sitting there waiting to wake you up.

It is always that balance, between - if you go back to work the next day, and you're like, "Oh, do I really try to fix this thing?" It could take a week to do. Or just hope that it doesn't happen again for a while. I patched it up and moved on. Then the busier you get, the less free moments you have to actually fix anything, and the more duct tape you get out, and the worse it gets.

Burnout is a really common problem still, across the industry. But especially for system administrators. We had the same thing at Yelp just a few years ago. Different story, but still the same thing. I mean it's still - sometimes amazing, that you see these really, really big websites where - sometimes you hear about the consequences, like, "Wow - if this thing goes down, it'll cost us like millions of dollars." They're still being run by skeleton crew sometimes. Often crews that aren't 24/7.

I can remember back in the first dot com boom, back in 2000 - I was at the company Xdrive. We had a 24/7 knock. We had like a network operation center. We had humans sitting there 24 hours a day, just in shifts. We would hire young people straight out of school, or people without a degree or something. They would just sit there and watch things. They would call somebody else if there was a problem or something, and they needed to escalate - but we had people there. It feels like nobody does it anymore. You're just relying on people to sacrifice their weekends and nights, to keep these websites up and running.

Len: It's interesting you mention that. I think it wasn't too long ago, that there was a big Facebook outage or something like that. It wasn't like this, but it was the technical equivalent of someone tripped over a cord, or something like that. People were very surprised that there's still these kludgy vulnerabilities built into even the biggest and most famous websites.

Just moving on a little bit. You had these very intense - and I imagine, wonderful years in New Zealand. But you eventually decided to move to London. I'm curious - having moved to London myself a couple of times, what neighborhood did you choose to move to when you first moved there?

Matt: Wow, I know, it's so controversial when you move here, which part of London you move to. I was just telling somebody this the other day, that - we asked everybody we knew that had lived in London. Because a lot of Kiwis do a tour of duty in London at some point. They call it like the "OE," the "overseas experience." Usually they take a gap in-between university and their first job in New Zealand or something, and go to London for a couple of years. But honestly, ifyou would ask 20 people, "What part of London is the best part to live in?" you may just get 20 different answers. We never heard from anybody twice.

Right in the beginning, we ended up in Wapping, which is right down by the river, near the Tower of London. It's the old historic docklands, which was really cool. We liked it there - but after a year or so, we moved to Islington. We live in Highbury now. I live right near Arsenal Stadium. When the games are on at the Emirates, I can hear it through the window.

Len: That's funny - I used to live on City Road, not too far from Angel Station, when I was working in the City. I could hear the stadium as well from there. It's funny - you said it is very controversial, people can get very defensive of their neighborhoods and where they live. But part of the adventure of living in London, can be moving around from different parts of it as well, and seeing how different life is in different parts of it. When I first moved there - it was in 1999, and I lived in Balham. I gather Balham is upscale now, but it was not at that time. There was no coffee shop in Balham at the time. There were no Starbucks or anything like that, but - yeah, so, well that's really great.

And so then you worked for Yelp, and you worked for Lyft. Were you in similar roles there?

Matt: Yeah. When I first moved, I still had a foot in the VFX industry. I was working for the startup called Avere, that did network storage. We were a big customer of theirs at Weta. They needed somebody in Europe on the technical sales side to go around. That was really fun. I did that for a couple of years, and I got to go meet everybody in the VFX world in Europe.

But eventually I decided that I wanted to get back into people management. There was something about that I really missed. I switched and went to work for Yelp, who had a smaller engineering office. They're San Francisco-based. But one of the - I think the VP of Engineering or the CTO or something, was British, and had this network of British techs that he had worked with. So he started up an office in London, which was really trying to solve this problem that we were just talking about, in terms of like, how you run a 24/7 website without keeping people in.

Well, London is eight hours away from San Francisco. You park a team of sysadmins in London, and now you have 16 hours of the day covered while people are sitting in front of their screens. I guess the dream would always be maybe to put another team in Sydney or something, and cover the third shift. But, anyway, that was the idea.

At that point, when you were on call, you were really only expected to cover like 12 hours of the day, instead of 24 hours. Somebody else, in the other time zone, would cover the other part. That was what the office was started for.

But then once it was there, it became this magnet where you started building up different teams, mostly on the infrastructure side, but all kinds of different database teams and streaming teams. All kinds of different things.

Len: Was it similar at Lyft as well? Did they open up a London office, partly for the time zone reasons?

Matt: No, not at all.

Matt: That's a completely different one, yeah.

Len: Okay.

Matt: Lyft was completely based in San Francisco. But the self-driving division was based in Silicon Valley in Palo Alto. Then they acquired a computer vision startup in London. There was a small tech startup who was doing some really cool stuff with mapping and computer vision. So they acquired them, and integrated them into Level 5. That's how they ended up with a London office, yeah.

Len: Just before we move on to talk about your book in the next part of the interview, I just wanted to ask you briefly - this has become a little thing we've been doing for almost two years now, unfortunately - asking people about how the pandemic's currently affecting them in their life and work. We get to interview people from all around the world, and you're probably the fifth or sixth person in London in the last year that we've interviewed. If you wouldn't mind, what's it like for you now?

Matt: it's okay. At the moment we're right in the middle of the Omicron wave, where it just feels like everybody in London has it. I don't know how I haven't gotten it. I mean, I'm pretty cautious. I'm much more cautious, I would say, than maybe the average person. I'm in a position where I'm able to do that.

It's really a position of privilege these days. If I worked at a supermarket or something, I wouldn't have these choices available to me. I'd have to go out into the community every day, and I'm really aware of that. My wife works with NHS, and she works in a lab. She's had to work throughout the pandemic, going in every day, because she can't take her work home with her. Obviously, when you're working in a lab, that's not available.

I've been working from home. We sent everybody home, I guess like everybody else, in March, 2020. I didn't go back to the office, I think, until August this year, was my first time setting foot in the, August of 2021, my first time setting foot back in. But we didn't really go back in. Now of course, we're all strictly working from home again.

So, yeah, it's been interesting. We still have tons of friends and family back in New Zealand, who we talk to regularly. Who are just living this completely different experience of the pandemic.

It feels like, now that they have cases and everything else, it's like we're living in the future here, and talking back to other people like, "We know what's coming. Watch out. We know what it's going to be like." We've had, I think a little bit more exposure to that side of things, just through family in New Zealand.

Len: One thing I'm just curious about is, are people wearing masks outside?

Matt: No.

Len: Okay.

Matt: No, not at all. It was never really a thing in London. The government never mandated it. And so nobody's ever really picked up on that.

Len: It's curious, where I live on Vancouver Island in the City of Victoria, that never became really a thing here. Occasionally when there's spikes of concern in the news, I've noticed that I might see a few people with a mask on outside, that's definitely happening now a bit. But it never really became the conventional thing to do here either.

Thanks very much for sharing that. Unfortunately it's been, yeah, since March, 2020. Hopefully I get to stop asking this question at some point.

Matt: Yeah. One of the things, just to start thinking about the book, was, the big push I had to getting the core of it done, because I've been working on it for a few years now, was, I took advantage. One of my ex-colleagues from Yelp, [Flavian], who's a really good Rust programmer, and has always been super helpful and encouraging, and everything else - I think like May or June or something last year, he moved to Australia. He had to do a two-week quarantine in a hotel in Australia. So, I took advantage of that - cracked the book, and get the core of it done.

I said, "Well, you're going to be locked up in a quarantine hotel for a couple of weeks, do you want to help me review the book?" I took time off work here, and he was locked up in a quarantine hotel in Australia. We just went back and forth across those time zones. He would review the book all day, and I would wake up, see all his notes, fix everything up, address all his concerns and everything, and then leave him a bunch of stuff to do for the next day. Those couple of weeks really were the most productive time I ever had over the whole three years of writing the book.

Len: Thanks very much for that, that's really interesting to hear about the multi-hemisphere editing process that's going on. And thanks also for the segue into talking about the book, Rust From the Ground Up

I was wondering if you could talk a little bit about, for anyone listening who might not know, what the Rust programming language is, and maybe what it's primarily preferred to be used for in programming?

Matt: It's one of the newer programming languages. I guess it's probably about ten years old at this point. It came out of Mozilla, the company behind the Firefox browser. They had some programming language researchers on staff who were thinking about this stuff.

Of course, Firefox is a big pile of C++. Which, inevitably, leads to different security problems or stability problems, crashes. I think their motivation was really around security. Because of course, the browser is the way everyone connects to everything these days.

When you have a hole in the browser, it effects everything. They developed this language. It's really designed for systems programming, which is lower-level programming, and really aimed at the market that C and C++ are in now. So it's up and coming and really interesting.

Over the course of my career in systems, I did have some gigs where I was full-time programming. But often it's a mix. You start dipping your toe into systems programming when you're running these systems.

Because you say, like, "Oh, there's some bug in Apache," or something like that. "Nobody else has fixed it yet. But it's affecting me, so I'm just going to wade into the source code and try and figure it out.".

The first time you do it, it's maybe, you fix like one line of code, and you patch it. But then it goes from there, and you start writing more and more. You say, "Well, it'd be great if we had a little tool that did this."

So, I ended up writing some network monitoring tools in C. Eventually some NFS tools that we were using to monitor all the NFS storage at Weta. I wrote those in C.

Eventually I had the idea, like I think with most C programs, it was like, "Oh, it would be good if this was multi-threaded." I think I was trying to do something where I could connect to multiple servers at the same time? I said, "I hear everybody's doing this stuff in Rust now, so maybe just rewrite this existing program that I wrote? I'll rewrite it from C into Rust."

So I waded into that, and just realized I was way in over my head. Rust is a completely different beast. You get a lot from it. It gives you a ton of guarantees about the security and quality of the code that you get out.

But the learning curve is super, super steep, especially coming from someone with a C programming background. So I just found myself like, "Gosh, I just have no idea what I'm doing." Here I have this multi-thousand line code base that I'm trying to mess with. I said, "Let me start with something simple."

I'd spent a whole bunch of my middle part of my career working with the BSD systems, which are somewhere between - Solaris was like the big commercial Unix that I started on. Then Linux is the big open source thing. But BSDs are another evolutionary branch of Unix. I spent a lot of time in the free BSD and open BSD worlds, so I was pretty familiar with those.

Len: Just for people who might not know, that's the Berkeley Software Distribution.

Matt: Yeah, that's right. It goes back to the early 80s. Some people at Berkeley, including Bill Joy, who's a famous programmer, started doing open source contributions on top of Unix. It eventually became it's own open source operating system. They're still around, free BSD and open BSD are still really, really going strong as an alternative to Linux. And in fact, a lot of the early, the Linux tools were rewrites of the BSD stuff in the early days.

I just had this idea of like, "Let me just start with really, really simple programs like when we write." There's a program called Cat, which is short for "concatenate." It glues files together. Although, a lot of times people just use it just to read a file out. But it's really simple. It's like 100 and something lines of code.

At Yelp we had hackathons, I think three times a year? Where everybody would get basically three or four days to just hack. We would put all project work on pause. You could just do anything you wanted to do. It didn't have to be work-related.

It could be work related. You could say, "There's this bug in some system that I'm not responsible for, that I've always wanted to fix. My manager would never give me time to do it. But now I don't have to ask anybody's permission, I can just go and fix it."

Or people would do physical projects. There was a lot of people bringing in Raspberry Pis and doing stuff like that. It was just a chance to express yourself. I think I was spending all my time in meetings as a manager, and not really doing any technical work.

I used to love hackathons. Because it's like, there used to be a sofa that was designated as my sofa, that I would just sit on during hackathons. I'd put my headphones on, and I would just have my laptop, and I would just code, like I was in college, right? You just code for like ten, twelve hours a day.

That's how I started to teach myself Rust. I took a lot of notes as I was going. Just like a Google Doc or something. Just for my own benefit, I was just like, "Oh wow, I just found this thing. Here's the sharp edge, here's the thing." Like, "I had to do some research to find this out." I was just taking it.

At a certain point, I thought, like, "Oh, this might be an interesting series of blog posts." Then through some conversations with some other people at work, it was maybe the germ of a book had started forming.

So I said, "Instead of a series of blog posts, why don't I do a book?" That's the genesis of the book.

The idea is that in every chapter, I take a Unix utility from the old BSD sources. In fact, to keep things simple, I'm sometimes going back into the 80s or early 90s, just to keep the core program quite simple. All these programs have grown in complexity over the years as people keep adding features. But the original ones are usually quite simple. So I just do one per chapter. We just rewrite it from BSDs C code into Rust. I'm teaching it as I go.

Len: Thanks very much for sharing that story. So you're publishing it in progress, if I have that correct?

Matt: Yeah, that's right. This is how I found Leanpub. I think I heard about it on Hacker News? Which was great. It was like this escape valve. Because I'd been working on it for maybe a couple of years. Trying to get to the end. There's never really a point maybe at which you're done. But I had this vision of how many chapters I wanted to do, and which programs I wanted to write, and everything else. But it was always like so far in a horizon, it was really difficult to stay motivated. And then I started hearing about stuff like Leanpub, where it's like, "Oh, I don't have to have the book finished. I can do it as I go." I'd basically gone through -

The other thing is, I'm teaching myself Rust. I wasn't a Rust programmer at the start of the book. I'd consider myself one now. Although, probably not the best one ever. But I've obviously spent a couple of years staring at this stuff. I wrote myself all the way through the book, several times, at this point. Because I knew you'd get to the end of the book, and then you're like, "Wow, I know all this stuff now." Now I can go back to chapter one, and you look at the stuff you wrote in chapter one, and you're like, "This is terrible." I had no idea what I was doing.

I've re-written the whole thing, I've lost track of how many times. Because it's this process of, you write chapter one, and then you write chapter two. In the process of writing chapter two, you discover some things. Then you go back and you rewrite chapter one. Then eventually you finish that process. Then you write chapter three. Then as you write chapter three, you learn some new stuff. You go back and you rewrite chapter one and chapter two. Then you do chapter four, and you rewrite one, two and three. Every chapter leads to this kind of thing.

But yeah, I have like the whole book written. I got to like 300 pages, or something like that. Now what I'm doing is going back and editing and typesetting and everything, each chapter, one at a time. So I published the first three, and I'm trying to finish number four now.

Len: That's very gratifying to hear that you're using Leanpub that way. That's basically the reason Leanpub exists, is for book projects like that. Originally, the idea of like, "Why would you rewrite a published chapter?" might have seemed unusual to people. But if you're writing about an evolving technology, and more specifically, something you're evolving on, then of course you're going to rewrite an already published chapter, to make it better or correct mistakes and things like that.

From the other side of it, from the reader's side, if someone else is learning Rust for the first time, they don't care if your chapter one is perfect. They desperately want that content. They want to get started and learn. Then if they run into a problem that you fix in the chapter, they're overjoyed that the chapter they already read was edited and is there for them to read again.

Matt: Yeah, that's right.

Len: That's really great to hear. Have you been getting any feedback or soliciting feedback from readers?

Matt: I haven't been soliciting feedback. I've soft-launched it. This is definitely going to be the widest audience I think that's been exposed to it. Because it also just felt like, again, like I don't have the sort of - when I published like the first version, I think I published the first two chapters at once. Because the first one is pretty short. But it didn't feel like I had anything really, this huge thing, to make like a big fanfare about. I was like, "Oh, let me get a few more done." I'm trickling out word as I go.

I do a little bit on Twitter now. But nobody knows about the book, so they're not really following me on Twitter yet. I mention it sometimes on Hacker News. But I haven't done like a big Show Hacker News, because it doesn't feel like I have - I'll save that for maybe when more of the book is complete, if not the whole thing. But maybe at least get over halfway or something? So something is trickling in, but it is actually really good.

I wasn't really planning on announcing anything, but somebody on Hacker News asked a question, like, "Hey, is anybody writing a book that teaches systems programming by doing one program at a time?" Or something like that - it was like exactly what I'm doing.

So I was like, "Alright, here we go." I wrote a comment and said, "Yeah, it's up on Leanpub. Here it is." I think I got about my first ten sales out of that. Then one of the people that bought it, sent me this really detailed email. Basically with all the things that I had gotten wrong. It took me a little, a day or something to absorb that, right? To be honest, I think, like, with most people, it's hard to put your project and your baby out there in the world. Because you know that the negative stuff is going to come in.

I think that was partially behind my motivation to get a lot of it done. Because I thought if I published it literally as I was writing each chapter, without having more of the book done ahead of me, and I got a whole bunch of negative feedback, it might be so demotivating that I would just give up. I was like, "I want to get the thing done, then put it out there. Then if people say stuff, it's like, 'Well it's already done, I just need to tweak it,' and I can power through."

But somebody sent me these really detailed notes. So, when I went to publish the next chapter, I wrote back to him and I said, "I've incorporated the changes that you suggested. Could I send you the next chapter before I publish it and have you copy edit it?" He really kindly agreed to do that. I found an editor just organically through this process. Because I wasn't planning on hiring an editor or anything.

But I really have, yeah, this person, Dan Wilhelm, who is doing a great job. I'm trying to be really respectful with his time - he's not a full-time editor or anything, but he provides this really direct and great feedback that's really improving the quality. I really appreciate it. But it's great how this all happened, just through the community around us.

Len: That's a really great story. That reminds me of a very old Leanpub story from a long time ago. It's a true story, but a legend, I guess I should say? But years and years ago, someone published a book on Leanpub, and then discovered someone had translated it into Russian, and basically published it on their own. Instead of getting mad, the author contacted their independent -

Matt: Unofficial translator, yeah.

Len: The independently-appointed translator, and said, "Hey, why don't you translate my next book, and we'll share the revenue from the translation?" The person, of course, agreed to that, and became an official translator.

Matt: Oh wow.

Len: As opposed to a pirate translator. It is amazing how people can come together on projects like this. The person's motivation wasn't to steal, it was to have a Russian version out there, for people to read in Russian. Because they liked the book, and thought it was important that people be able to know it. Perhaps, they were just too shy to reach out, or something like that? It is just amazing how people can come together around projects like this, even sometimes when it's totally unexpected.

The last question I always ask people on the podcast, if they're using Leanpub, is, if there was one feature we could build for you, or if there was one really annoying thing about Leanpub that we could fix for you, can you think of anything you would ask us to do? Other than improve Google search?

Matt: Yeah, yeah, yeah. That's definitely out of your control.

Len: Yeah.

Matt: In the big machine over there. I'm trying to think. Has there been anything so far that I've run into? I guess, as I do, maybe the steps are there somewhere, and so maybe this has just turned into a tech support call. But as I publish new chapters, it's not really clear to me how many of the original folk are downloading those. As I go, it's really great, you get these emails, especially in the middle of the night, you get an email, "Hey, somebody bought your book," which is really super gratifying. Then you publish a fresh chapter, and I guess all the existing people get an email saying like, "Hey, there's an updated version of the book." But you don't get that gratifying thing of like, "Oh hey, all these people that already bought the book just like downloaded the new chapter." You just get that initial hit of adrenaline when they first buy it.

Len: Thanks so much for sharing that. Actually, I mean, and by the way, we save this part of the conversation for the very end, because sometimes it does turn into tech support. Which is great, from our perspective. There actually are people who skip the first parts of the interview, and go to the end, where it's like all about writing and publishing and stuff like that. And - one of the reasons, if you find a feature on Leanpub that works, it's probably there because some author asked for it at some point.

When I say, "if it works," that's maybe because an author found some technical problem with it, or something like that, and, we've been blessed that a lot of our authors are actually programmers themselves. They're people who are used to finding, reporting, and fixing problems.

With respect to what you're talking about, that's really interesting that we don't - what we don't do is give you the dopamine hit of "someone's downloaded your book." You can toggle on and off "email me if," and "notify me." But we do have a page called, "Download Formats," that does give you some statistics about how many people downloaded the PDF. Or not how many, what proportion have downloaded PDF, EPUB and MOBI, and stuff like that.

Matt: That's what I found. But I have 100%, I only publish a PDF at this point.

Len: Oh right, of course. Yeah, so that's not useful.

Matt: That's like 100%.

Len: I'll think about that. Because that - I mean one of the reasons Leanpub's in this in-progress publishing business, is that, it's for motivation. You get your book out there early, start getting sales. Things like that. And, yeah, letting people know, "Oh this," Now, all of a sudden, "100 people have downloaded it," or something like that. That would actually be really good for authors to be able to get that gratification. Or to know if no one's downloading it, and, "Is there something else I can do about my messaging?" Or something like that. That's a really great suggestion. I'll make sure to note that for the team.

Well, thanks very much for sharing so much of your story with us today.

Matt: Yeah, not at all.

Len: And also the story of how you're writing your book. Best of luck with the project, and thanks very much for being on the Frontmatter Podcast.

Matt: Okay, anytime. Thanks for so much for building Leanpub.

Len: Thanks.

And as always, thanks to all of you for listening to this episode of the Frontmatter podcast. If you like what you heard, please rate and review it wherever you found it, and if you'd like to be a Leanpub author, please visit our website at leanpub.com.