It's all fun and games until someone [XOFF]

It's all fun and games until someone [XOFF]

Way back in the dark ages of Linux, I had a bunch of machines which didn't run X. They were strictly text mode, and sat there doing whatever I needed them to do: routing, DNS, dialups, mail, RADIUS auth, you name it. There were plenty of daemons working for me, and most of them had things to say via the syslog.

To keep an eye on things, I would just watch the syslog by tailing files like /var/log/messages. That was all well and good, but it meant having to be logged in. If I was logged in, then there was a console open, just a single ^C away from giving a shell to anyone who came by. This was before the days of "cheesing", "Biebering", or "jelloing" unlocked terminals, but I still didn't like it.

One day I noticed that you could create a new text console on a Linux box just by shoving some data at the right /dev entry. Just "echo" something to /dev/tty12, then ALT-F12 and you'd be able to see it. Prior to that, ALT-F12 would do nothing since it didn't exist yet.

From that discovery it was just a hop, skip and a jump to having syslogd write a second copy of everything to /dev/tty12. Then I could log out, flip to that virtual console, and watch things that way. Any time I wondered what was happening, a quick tap on the keyboard to turn the screen back on would let me see without logging in.

That's how it went for a while, but then one day basically everything on that machine stopped responding. It's been a long time so I forget exactly transpired in the middle, but eventually it came down to one thing: my keyboard's scroll lock light was glowing at me when I was on tty12.

Yep, that key that basically has been a part of the PC keyboard layout forever and never really did anything finally had a purpose, and on Linux, it served to stop writes to a text console. This was all well and good in theory, but since syslogd was doing blocking writes to it, that also meant syslogd would get jammed. These days, that wouldn't be a problem, but back then, /dev/log was connection-based.

When syslogd got stuck, it stopped reading /dev/log, and eventually that became a trap, too. Anything trying to talk to it also blocked. Given that sendmail and a bunch of other things all called syslog(), this made for a pretty messy situation.

Someone had pressed the scroll lock key on the keyboard while that tty was active. Maybe it was me, or maybe it was something fooling around, or maybe it was just the cleaning crew "doing me a favor". It doesn't really matter.

What inspired me to write about this today is stumbling over something similar not too long ago. I have a couple of programs which run in screen, mostly out of sheer laziness on my part. This is fine until you accidentally hit ^S while attached to it. ^S is XOFF, or for those of you who are lucky enough to not have to know this... is part of software flow control.

Yep, ^S, 0x13, decimal 19, DC3, whatever you want to call it, is usually interpreted as XOFF. It'll sit there and block writes until someone sends it ^Q, 0x11, decimal 17, DC1, which is XON. Really.

Anyway, let's say you're in screen, which by default uses ^A as a command key. ^A N goes to the next screen, ^A D detaches, and so on. If you're on a vaguely QWERTY-ish layout, you'll notice that A and S are right next to each other.

It's not much of a stretch to imagine accidentally hitting that S instead of A. At that point, all of the writes block until you either unwedge it with ^Q or bail out of screen. If you didn't know what was going on behind the scenes, it might just seem like "one of those things that happens sometimes".

If you've ever wondered why some folks hit ^Q any time things seem to have frozen in their terminal, this might be why.