This all started when I tried installing snapd on an Ubuntu 14.04 machine using kernel 3.13.0-170-generic. As part of the install process, the kernel was updated to 4.4.0-148-generic. After the install, I rebooted. Now my Django site stopped working.

It turns out postgresql was no longer successfully starting on boot up. Also, htop showed that both my CPUs were maxed out. The PIDs of the gunicorn processes kept changing. It looks like they were repeatedly starting, crashing and restarting. When I used supervisorctl to stop them, my CPU use dropped dramatically.

When I tried restarting postgresql, I got this error:

sudo su postgres
service postgresql start
Failed to issue method call: Unit postgresql-9.6.service failed to load: No such file or directory. See system logs and 'systemctl status postgresql-9.6.service' for details.

I checked every log I could think of, but there were no clues. Eventually I ran ps on my production server to find the command that launched postgresql. When I ran that I got:

could not open temporary statistics file "/var/run/postgresql/9.6-main.pg_stat_tmp/global.tmp": No such file or directory

Googling showed lots of work arounds. The most promising was to write my own script to create that missing dir during startup. However, I have another server running the same project. It did not have any special files for creating that tmp dir. So instead uninstalled snapd and updated all my packages. Now everything was working.

I still wanted snapd, so I reinstalled it and rebooted. Postgres stopped installing on boot up. Gunicorn was thrashing. I stopped gunicorn. Started postgresql. And restarted gunicorn. Everything was working again. Reboot broke it again.

I disabled gunicorn startup on boot. After reboot, gunicorn was not running and postgresql was not running and htop was not maxed. I could manually start postgresql.

After running down many rabbit holes, I could not solve this problem. So I removed snapd again and updated my packages again. Things were working again.

I decided that rather than continue to work on this out-dated solution, I will work on upgrading my entire OS. I have to do that anyway. I suspect most of these problems will go away after that.