Asking nicely for root command execution (and getting it)

Asking nicely for root command execution (and getting it)

There was a SEV review meeting once upon a time, and in it, we had reviewed some incident where something bad happened involving something that ran as root. I don't remember the finer points of that one any more, but the resident VP who ran the show wondered aloud how much "root exposure" we had in our infra in general. That got my attention.

I decided to try to get an answer to it. While in the meeting, I hit up my little dataset of everything running on any machine in production, then narrowed it down to anything running as root (that is, uid 0), and also with some (TCP) network listening ports. I figured those would be the easiest to "pop". Think "SELECT ... FROM ... WHERE uid = 0 AND ports IS NOT NULL" type of thing.

After the meeting ended without using the whole 90 minutes, the room was still reserved for another 15-20 minutes or so, and a couple of people hung out to look at the list. One noticed that this one service running everywhere had these options for "pre" and "post" commands. This was a service which was normally used for performance measurement stuff and involved running an external command. You'd say "okay, analyze this thing for me", and it'd crank for a bit and kick back a bunch of data points on whatever you had targeted. That much was fine, but unfortunately you could specify arbitrary commands to run before or after the request.

One of the people there came up with a test and built a command that should run "touch /tmp/(their name)-was-here". Then they fired it off, and another person looked, and sure enough, the file had appeared in /tmp, owned by root. (I should note that this person didn't have any magic permissions for that service, lest you think that's what happened here.)

This was a pretty scary hole: you could ask it to run anything, and those commands would be run as root. Uh, sigh, yeah, that turned into a security SEV right there on the spot. Every damn machine running that software could be owned by anyone who sent a properly-formatted request to the service on the box.

Also, there was no log of this request. The person's name was the sort of thing that was easily grepped, and it didn't turn up anywhere. That means it didn't log the actual command run (since it would have contained it), and it didn't log the request which arrived over the network (which could have contained their username, but didn't).

So now this meant we had a root hole basically everywhere, and no way to find out if it was ever exploited. It was a terrible situation. Then the next step was to go back in the source code and find out when the pre/post feature was added, when it shipped to production, and then do the date math, to find out how long things were vulnerable. When it turned out that it had been there for years, well, now we had no idea whether someone else found it, exploited it, and shoved off long ago, all with zero detection.

What else could happen to make it worse? Well, we found out that there's Actual Business Stuff using this pre/post feature, and so we couldn't just turn it off pending a fix, at least, not without breaking that "Stuff". Fun, right?

Fortunately, when this happened, one of the people on the service's team was available and responsive, and understood exactly how to thread the needle of closing the hole and not breaking the business. They built a list of every pre/post command that had been in actual use (by looking through the source tree for clients of that service, naturally), and then turned it into a hard-coded check list. The program would now refuse any pre/post commands unless it was one of those already known to it.

This change patched the hole and so was cherry-picked onto the last release and was pushed out rather quickly, and that was the end of that round of fun. Unfortunately, it was just the beginning of the general pattern of "oh hey, I bet I can pop that thing", as a great many more were about to be found.

You'd think this would have touched off a new "golden age" of people being thoughtful about not running things as root unless absolutely necessary, and generally being paranoid about handling subcommands, but that's not how it panned out in general. Instead, I'm pretty sure it pissed off the wrong people, and the rest is pretty much as you would expect.

Suffice it to say, if you work someplace with enough machines, there's probably some way for you to get root on all of them if you can hit them with a handful of packets. I've seen it happen far too many times at enough companies to expect things to stay secure. I'm not talking about buffer overflows and stuff like that, although those exist too. I mean just straight up asking a service to please run a command for you (as root), and it gladly complies.

Maybe this is our version of the "infinite monkeys" thing: given enough software people, enough computers, and enough time, someone at a company will eventually grant universal remote root access to anyone who knows how to read some source code.