Recovering from a lost disk: how I setup, backup, and restore dev machines

Monday, July 24, 2023

Last Wednesday just before 3pm, I went pack up my laptop to get ready to drive 7 hours to visit my family in Ohio. Fedora had some updates to apply and when it went to come back on after those, I saw the words no one wants to see:

Default Boot Device Missing or Boot Failed.
Insert Recovery Media and Hit any key

Panic sets in, because this is my main machine and it's not coming on. I tried a few things but long story short, it's dead as a doornail. After we got in that night, I confirmed that the SSD was dead and the motherboard was fine. I guess that's good.

The next day I got to start the recovery process. Well, when life gives you lemons, make lemonade ✨ content ✨. My recovery process was pretty smooth, and this post talks about how I setup machines to make it painless to setup a new one. I'll cover backups and restoration first, then my dev environment setup, then some odds and ends that make life easier.

Disaster averted because of backups

Fortunately when my SSD died, I had relatively recent backups. I use restic and have had a great experience with it. My backups are stored, encrypted, on Backblaze B2, and I run them manually1 each week.

The only stuff I worry about backing up is files in my home directory. Anything outside of that on my personal machines is disposable, and programs are installed separately in my dev environment setup (next section). Doing it this way means when I need to recover from a dead machine, I can easily pull down all the files I care about and be back up and running in just how long that download takes.

The script I use weekly is straightforward:

#!/bin/bash

source env.sh
sudo -E restic -r b2:$BUCKET:$REPO --verbose backup --exclude-caches --exclude-file=./.restic-exclude /home/nicole

env.sh is another script which looks like this:

#!/bin/bash

export BUCKET=XXX
export REPO=XXX
export B2_ACCOUNT_ID=XXX
export B2_ACCOUNT_KEY=XXX
export RESTIC_PASSWORD=XXX

This is a bash script instead of something like a .env file so that I can use it without any dependencies on the system. The goal here is disaster recovery where no tools are available yet on the new system.

The main backup script runs a restic command, which we can break down piece by piece:

• sudo -E runs as root while inheriting the environment variables, so it will get the B2_ACCOUNT_ID and whatnot from env.sh
• restic -r <location> --verbose backup <options> /home/nicole does the backup itself, with some more options added on to exclude caches and whatnot, and then specify my home directory

That's all there is to it!

To restore, I created a temporary directory and ran:

restic -r <location> --verbose=3 restore -t Restored/ <snapshot-hash>

It was fairly straightforward, but took a long time. My one gripe with restic is that restores cannot be resumed if interrupted. Mine was interrupted because I'd set a low threshold on daily spend limit for my B2 bucket, which I hit when 3/4 done downloading my restore. I had to then wait for a new cap to take effect, then redo the entire restore on a relatively slow internet connection. It worked, but wouldn't be tenable if I had a flaky internet connection.

If you don't have backups setup, make sure you do so! It makes the whole disaster recovery process less stressful knowing that my data is able to be restored. Also make sure you regularly test your backups; I'd not done a restore before this one, and now I know to work it into my routine.

Scripting the setup of my dev environment

Every developer has their own way of setting up their local environment. Some people do it manually each time, making each machine a bespoke experience. Others go full bore and use devops automation tooling, like Ansible, to manage their dev machines. I'm somewhere in the middle with managed chaos.

My config repo is open source (AGPL) and is the same repo I've used to store my config files for all my dev machines since 2011. Its organization has changed a bit as I've evolved how I do things, but now it comes down to a couple of bash scripts and a pile of config files that I link into the right spots.

The bash scripts are straightforward. I start with a bootstrap script and then run a config script.

The job of the bootstrap script is to install the essential programs that I need for daily life as a software engineer. I used to make this full of conditionals so that I could rerun it. Now the one I use (fedora_bootstrap.sh in the repo) is mostly just a couple of dnf installs, installing rustup, and installing other tools like tmuxinator. This one changes each time I run it; I keep it simple and tweak it based on what I want on each machine. It's easier to just edit it each time than make a more complicated config system, although the itch is there... convince me!

The second script is the meat of the actual config work for all my scripts. The config files (or "dotfiles" in dev parlance) are stored in their respective directories. Neovim configuration is in ./nvim/, my bashrc and profile are in ./bash/, etc. To install these, I have a config.sh script which uses stow to link them into my home directory:

#!/bin/bash

stow -t ~ bash
stow -t ~ git
stow -t ~ tmux
stow -t ~ nvim
stow -t ~ editorconfig
stow -t ~/.config/ tmuxinator

And like that, the config files are all in place!

After setting those up, I have to go through and do things like install my neovim plugins. This is a manual process but a very easy one (run :PlugUpdate once), so I haven't had the urge to automate it yet since I don't have to do this very often. It could be neat, though, especially to do it in an idempotent way!

That's really all I have to it. Since my setup is pretty light, and it's all in git, I check out the repo and do this stuff. It's very empowering to be able to quickly, effortlessly spin up a new dev environment!

Other quality-of-life things

Now there are also some other programs I setup that aren't explicitly my dev environment (I wouldn't install these on a remote server) but which are handy for my quality of life when working on projects.

The first program I love here is Tactile, which is a Gnome extension that lets you easily resize windows to certain portions of the screen. I like the idea of tiling window managers but I've never managed to switch to one and I like to do minimal configuration on my machines. So this is a nice middle ground. It lets me use Gnome but easily resize things and tile them.

Next up is an email and calendar client. I am prone to getting distracted by every little thing, so having a dedicated mail client (currently Thunderbird) lets me refer to emails and my calendar without the pit of distractions that is a web browser.

I also use Obsidian for note-taking on my personal projects, so that gets installed as well. I use it in a fairly naive way, keeping daily notes and having a poorly organized personal wiki, but it works well and gives me a place to organize a chaotic mess of thoughts. Can't do without it now!

And of course, my password manager (currently 1Password) also has to be installed. This is essential for everything in life now. If you don't use one, get one. It makes you more secure and makes things more convenient.

Oh, final thing: I set Comic Mono as my terminal font, and Ayu Light as the color scheme. These are manual processes which I should automate someday. I started using the font as a joke but unironically love it and believe it's the best coding font out there. I'm reasonably happy with Ayu Light as the color scheme but would welcome suggestions for other light mode color schemes!

If you do anything differently, I'd love to hear about what you do and why! I could also write about my specific choices of dev tools (nvim and which plugins, bash over zsh, tmux, etc.) if anyone is interested.

Now that this disaster recovery is done, I'm going to get back to, shall we say, my regularly scheduled programming.