Backported patches mean version numbers of variable utility

Backported patches mean version numbers of variable utility

Version numbers are everywhere, and yet, they can be misleading. The worst interactions I've had with them came from my life in the web hosting support business. The intersection of library versioning, operating system backports and internal versioning and relatively unsophisticated customer audits would occasionally make my life complicated.

It goes like this. First, you have well-known programs like Apache and OpenSSH. Then you have libraries used by those programs like OpenSSL, libcurl. There are also runtime environments like PHP or Java. All of them have their own version numbering schemes from their respective developers and/or corporate goons.

Then there are operating system vendors like Red Hat, and their products like RHEL (Red Hat Enterprise Linux). They tend to roll up a bunch of releases of Apache and OpenSSH and all of those into a given product of their own, and then call it "version 5" or "version 6" or whatever. They then have point releases along the way, so it might be 5.1 or 5.2 or 6.3 or 6.4. Between those, there are security updates which might come out at any time, and might even be applied without you realizing it if your system is configured to auto-apply patches.

The catch is they tend to keep the same base versions of everything for the entire lifetime of a given product: 5.x, 6.x, etc. RHEL 5, for example, is "still" running Linux 2.6.18. Of course, it's not the stock 2.6.18 which was released way back when. It's had a whole bunch of custom patches applied to add features they decided were important but which actually came out in some later kernel. They also apply security patches as flaws are found. It's still 2.6.18 at its core, and so all of those basic behaviors remain in place, but it slowly improves with time while still remaining more-or-less stable.

Normally, this is not a big deal. People find out about flaws, find out their release was affected, then go and look and find that their machine has already applied the new RPM build from upstream. That new build included a patch to whatever the problem was, and now it's no longer vulnerable. This continues until they retire the release, and that takes on the order of 10 years if you're willing to pay for that kind of stability.

It gets ugly when you have a customer who only sees the "external" version numbers and doesn't see the whole picture. They find out that their server is "running Linux 2.6.18" and flip out because there might have been a dozen security holes discovered in that since it came out.

We'd usually get tickets from people who had run some external auditing script against their server. It would see the "Apache/1.3.30" or whatever and would proclaim that it had bunches of vulnerabilities from that alone. It didn't actually check for whether they were still vulnerable or not. Every time someone did this, some poor tech like me would have to go through and explain exactly what was going on. It usually went something like this:

"It wants at least version X of program Y because of vulnerabilities A, B, and C. Red Hat patched A in build 1, B in build 15, and C in build 30. Your server has build 45, so you have patches for all of those problems even though it still displays as some version before X".

Basically, these programs would trip out at the results of the Server: line in a HTTP response and would miss the fact it came from a RPM that was in fact up to date.

Usually, I could grab one of these tickets, do the foot work and prove that everything was already patched, and they'd be happy. I'd also try to explain the situation to them so they didn't freak out nearly as much in the future.

Unfortunately, now and then, someone would take them literally and would start trying to get everything upgraded to meet their demands. That is, if the customer said the audit program wanted PHP version X, then the tech would try to build it manually on the machine. This would mean rebuilding a great many other things which relied on it, since a Red Hat type distribution is a huge mass of interlinked dependencies.

This is a non-trivial matter, and you really don't want people trying to do this on a whim. They usually get it wrong, and remember, it's probably completely unnecessary due to the backporting of patches I described earlier. There's another problem that's not quite as obviously which actually will make things worse down the road: taking them off the main line.

By that, I mean that once a server is running a custom build of some package, be it PHP or Apache or even the kernel, odds are, it's not going to ever run the "official" package of that program ever again. It'll probably just have that one-off build installed as the result of that ticket, and that build will sit there and rot until the machine is shut down... or cracked.

This is because when a customer demands and receives a special build, they usually aren't moved to a new package stream. Instead, they only get the one patch and then that's it forever. Actually moving the customer's machine to a new provider for that package would mean actually having a provider and "build master" who maintains such things, and that's well beyond the scope of what support techs usually get to do.

So, if you're one of these customers, be careful what you ask for. If you make some unreasonable request and they make a one-off build of some package, you run the risk of never receiving another automatic update for that program ever again. If you really need to do this, make sure they move you onto another maintained source of packages and don't just "rpm -Uvh blah" and leave it like that until the heat death of the universe.

Stuff like this is basically easy job security for sysadmins.