Connor's Blog

Connor's Blog

r/PBSOD is a fun subreddit all about posting pictures of broken kiosks and information/entertainment systems in public. It stands for Public Blue Screen of Death which is a Windowsism but Linux systems are posted there too.

If you spend enough time scrolling, you might notice two things:

1. There’s a scary number of these kiosks running networked Windows
2. Nearly all of the Linux kiosks are in a panicked state

A kernel panic is an unrecoverable error. It could be a damaged filesystem, bad hardware, a malfunctioning driver, a bitflip, or just a kernel bug. When this happens, a backtrace is printed and the system hangs forever. It won’t respond to ACPI shutdown events and there’s no processes running to report that it’s in a bad state.

These panicked kiosks we see are in limbo until they’re power cycled. Some might get power cycled daily, but I’d wager somebody usually needs to come along and do it.

This is kind of tragic because it can be prevented by adding one parameter to the boot command line.

panic=60

Or it can be set ephemerally with sysctl:

$ sysctl -w kernel.panic=60

Use this and the kernel will automatically reboot 60 seconds after a panic. It gives you enough time to snap a pic if you want to look into the issue, but gets the system back up and running shortly if you’re not around to fix it. Of course, if the hardware is indeed damaged or the filesystem is broken, this will just throw it into a boot loop, but you’re no worse off. If you’re building a kiosk, it’s hopefully obvious to avoid having any read-write mounted filesystems at all costs.

In fact, this is worth using on any production system you manage. It doesn’t matter where your system is – whether it’s in the cloud, on-premise, or inside a fridge – one day it will panic and this might just save you from being woken up at 1am.