What we need to take away from the XZ Backdoor (openSUSE News)
source link: https://lwn.net/Articles/969591/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
What we need to take away from the XZ Backdoor (openSUSE News)
Debian, as well as the other affected distributions like openSUSE are carrying a significant amount of downstream-only patches to essential open-source projects, like in this case OpenSSH. With hindsight, that should be another Heartbleed-level learning for the work of the distributions. These patches built the essential steps to embed the backdoor, and do not have the scrutiny that they likely would have received by the respective upstream maintainers. Whether you trust Linus Law or not, it was not even given a chance to chime in here. Upstream did not fail on the users, distributions failed on upstream and their users here.
(Log in to post comments)
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 15:24 UTC (Fri) by gwolf (subscriber, #14632) [Link]
It's good to have a rough understanding of the diff between previous and new version, but this was not an easy to spot backdoor. And I cannot blame the distributions for not finding it *earlier*.
And, I must add: This is a case where the staging mechanisms used by the different affected distributions (Debian with the unstable/testing/stable process, and the RedHat ecosystem with the progression between RawHide, Fedora, CentOS and RHEL) worked *exactly* as it should. The backdoor was found, yes, by mere chance, by an over-zealous developer worried about milliseconds worth of difference, and not by formal auditing processes. But it worked: It effectively prevented the backdoor from being deployed in live, production servers.
Of course, if people choose to run unstable / bleeding edge distributions... Expect bugs and warts!
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 16:12 UTC (Fri) by JoeBuck (subscriber, #2330) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 17:13 UTC (Fri) by Heretic_Blacksheep (subscriber, #169992) [Link]
The fact of the matter is that *both* upstream projects and distro package maintainers will have to be more diligent who and what they gate into their code bases and repositories in the future. It's the responsibility of both groups, not just one or the other, to make sure what's packaged is not just reasonably bug free, but also reasonably free of malicious code. The tools to do this at scale will have to evolve. If people don't like that, perhaps its time to re-evaluate career or hobby choices, because TLAs aren't going to play nice just because someone's an unpaid volunteer. If this results in less software out there, but higher quality review processes and slower releases, then that's just going to need to be done. Corporations and governments with their security on the line are going to have to stop freeloading. Both are extremely bad at investing in necessary infrastructure till it's crashing down around their ears. Individual programmers can't realistically go up against nation states without greatly expanded resources, including education about secure coding practices, and in the case of the general public, better technological education and accountability.
At some point heightened diligence isn't going to just be a good idea. It's likely to become the law in many, if not most, countries. Software won't be usable by organizational or government entities without proper vetting. Those vetting processes will also have to evolve because the current methods of certifications are far too arthritic to keep up with modern security landscapes.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 20:47 UTC (Fri) by kleptog (subscriber, #1183) [Link]
Attacking open-source this way is a high-risk, high-reward strategy. Sure, if you could backdoor ssh you'd be all set, but the evidence would be in public view for all time. All it takes is one person to stumble across it. There's no way to ever remove the backdoor from history. If it were really that common, we'd see the evidence all over the place. And we don't. This attack was found precisely because the source was publically available.
Add to the fact that the TLAs and their respective governments also use open-source software everywhere, so they'd be building backdoors into their own systems. (I'm guessing the targetting of DEB and RPM was to avoid custom government distributions being affected.)
On the other hand, backdooring closed-source software is much lower risk. The number of people who would ever see the code is much lower. When a product reaches end of life the code simply vanishes. Insert a backdoor in an NVidia/Broadcom/Dell driver, who would ever find out?
So sure, open-source developers and maintainers need to get better at vetting, but it's the corporations where the risk is. And I guess the tools to discover these kinds of attacks are also going to be developed as open-source because there's no direct benefit to any individual corporation. And how could you trust a closed source vetting tool not to be backdoored itself?
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 0:01 UTC (Sat) by LtWorf (subscriber, #124958) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 2:14 UTC (Sat) by alp (subscriber, #136414) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 21:52 UTC (Fri) by AdamW (subscriber, #48457) [Link]
It's important to note you cannot know this. It's a counterfactual. We don't live in the world where Andres didn't find the issue, so we don't know what would have happened next. Perhaps what would have happened next is somebody else would have found it half an hour later. Perhaps not. We don't *know*. You can't say you know that "the malicious code would still be there. Jian Tan would still be working its tendrils..." etc.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 18:20 UTC (Fri) by willy (subscriber, #9762) [Link]
I'm always disappointed by things like Debian bug 897426 where the packager washes their hands of the patch and wants me to talk to upstream directly.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 20:36 UTC (Fri) by WolfWings (subscriber, #56790) [Link]
Like I see nothing at all done wrong here by Debian, you just came barking up the wrong tree.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 21:17 UTC (Fri) by willy (subscriber, #9762) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 22:32 UTC (Fri) by zeha (subscriber, #61580) [Link]
This was very useful in a time when most software was developed by single entities, without public bug trackers, mailing lists, etc.
> It's the job of the maintainer to coordinate patches with upstream.
If the job is mostly to copy-paste data from one bug tracker into another, its pointless bureaucracy. Please just talk to upstream directly when it's easy.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 13:57 UTC (Sat) by elw (subscriber, #86388) [Link]
If you combine this with a system where distro maintainers are the ones working directly with upstream, or even a part of the upstream community, it allows a strong rapport to form between the upstream maintainers and those submitting issues upstream. This not only decreases the quantity of reported issues but increases the quality of what is reported, further improving upstream’s ability to focus on what matters.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 7:13 UTC (Mon) by calumapplepie (subscriber, #143655) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 16, 2024 16:10 UTC (Tue) by NYKevin (subscriber, #129325) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 12, 2024 22:09 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 10:42 UTC (Sat) by smcv (subscriber, #53363) [Link]
Right - as a downstream maintainer, if I forward someone else's patch, I will not necessarily be able to defend its design decisions when upstream queries them, because they weren't my design decisions and I don't necessarily know why they were made.
For relatively obvious bug fixes ("there's a double-free here, solution: don't free it the second time") that's less of a concern, but for bug fixes that involve tricky changes that could have been done a different way, it's better for the author to be the one talking to upstream, and doubly so for feature requests where there isn't yet any specific "correct" behaviour.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 17:51 UTC (Sat) by dilinger (subscriber, #2867) [Link]
I regularly do the same thing with chromium. I package it, but it's a complex piece of software with lots of things I've never used/touched. When someone reports a bug about their screenreader not working properly with it, I request they submit it upstream. If they can't, I'll do it for them, but I've never even used a screenreader. If upstream asks me any questions about the desktop environment, or how to reproduce the bug, I can't tell them. Instead, I'll just pass the message along to the bug reporter. That's silly, and introduces lag (and maybe I'm busy filing taxes and forget to forward the email). Similarly, I've never used widevine or done chromecasting, so I'm going to send those reporters upstream as well.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 15:38 UTC (Mon) by nim-nim (subscriber, #34454) [Link]
The value is common auditable build system that enabled to identify all the bits the malicious code touched and rebuild them in a few hours.
Just try to do the same without a distribution, I dare you.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 8:10 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 8:16 UTC (Sat) by mjg59 (subscriber, #23239) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 8:22 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
No, it is not.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 8:36 UTC (Sat) by mjg59 (subscriber, #23239) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 10:55 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 11:24 UTC (Sat) by pizza (subscriber, #46) [Link]
Yeah, so? What else are you supposed to do when upstream explicitly rejects your use case?
> They then have to be scrutinized with the same effort that the upstream project is.
What makes you say the various downstreams didn't carefully scrutinize this particular patch? This wasn't a matter of "different effort" so much as "different priorities". Indeed, the path of least effort was to abandon the use case that led to that patch to begin with.
> Responsibility is not a given, it has to be taken.
...What does this mean in the presence of an *actively hostile upstream*?
...Do *you* personally review every line of code of every software component in your system, prior to compiling and installing it yourself?
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 13:57 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
For the Sake of toning this debate down a bit: on behalf of my employer, my colleagues and our customers, we truly value the work the distributors are performing, in many specific cases and in general, and by "truly" i also mean in the economic sense.
However, nothing you can say will convince me that patching critically exposed software is fine. It's one of the most ugly technological debts, and if done properly, consequentially incurs massive project cost. I speak from experience.
In this particular instance, if openssh does not integrate with systemd-notify upstream, *and* if another integration is not possible, *and* if system service management is a must-have, then we have a very critical situation. It must be resolved, clearly and openly, and not be obscured by downstream patches.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 14:19 UTC (Sat) by bluca (subscriber, #118303) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 14:32 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 16:37 UTC (Sat) by bluca (subscriber, #118303) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 14:57 UTC (Sat) by Wol (subscriber, #4433) [Link]
No. They are saying this (bad) state of affairs is the only option available to them.
> However, nothing you can say will convince me that patching critically exposed software is fine. It's one of the most ugly technological debts, and if done properly, consequentially incurs massive project cost. I speak from experience.
Tough titty.
What you seem to be missing is that "patching critically exposed software" may be a NECESSITY. And if upstream *WON'T* talk (as seems to be the case here), then you are between the rock of "running security software with known vulnerabilities" and the hard place of forking.
Both of which, in your words, "incur massive project cost".
But hey, if it's fine in your world to order other people to talk to brick walls, expect them to assume it's fine for you to talk to a brick wall as well :-)
Cheers,
Wol
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 19:55 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
In this particular example, i see no such a necessity. At least the following alternatives come to mind:
1. The team developing the system service manager could have opted to provide a less intrusive, more compatible notification mechanism.
2. The switch of the distribution to said system service manager could have been postponed until a time where such apparently critical security concerns would have been resolved.
3. The patches could have been applied, but this should have lead to the conclusion that a re-implementation of the a service for the secure shell and secure file transfer protocols was required that would meet the demand of Linux environments using systemd service management.
3a. This could also have lead to an evaluation of the secure shell protocol being an adequate remoting utilitiy for such environments.
I am convinced that with enough expertise and good will, a different solution for notification could have been found that does not require shoehorning it into the service using a downstream patch to an upstream project that obviously considered this an unsupported scenario.
My main point of critique about the entirety of decision-making that lead up to this incident is that there is indication that reasonable security practices were neglected in favor of personal ambition, and if it just was about gaining the upper hand in factional disputes (systemd vs. systemd-is-evil, bsd vs. linux and others).
Here's a theory about it: We work in a very competitive industry (IT), we are constantly exposed to the internet with all its trolls and an unfiltered melange of patterns and anti-patterns, *and* we do F/LOSS, which makes us feel like in a hamster wheel, just that it's not about running, it's about arguing with morons. On top of that, many of us are badly funded and feel exploited, be it by megaclouds, tech billionaires or even our own users. It makes perfect sense that we are susceptible to factional infighting and zealotry. Sometimes, that mindset even might be adequate. But in IT security it is not, and while problems like stress and underfunding are all understandable, the rules of the game change into a direction where such practices will not be tolerable anymore.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 20:47 UTC (Sat) by kleptog (subscriber, #1183) [Link]
There are no other notification mechanisms, this is the only game in town. It's not even systemd specific. Why would we need a different notification method for OpenSSH than literally every other daemon? It's one line of code, hardly intrusive. Most other system daemons on Linux support notification out of the box.
> 2. The switch of the distribution to said system service manager could have been postponed until a time where such apparently critical security concerns would have been resolved.
The patch was one line of code. There was no indication there was any security problem with it. The reason it was rejected wasn't (AFAICT) security related. You're not going to halt a migration for something fixed by a one line of code.
> 3. The patches could have been applied, but this should have lead to the conclusion that a re-implementation of the a service for the secure shell and secure file transfer protocols was required that would meet the demand of Linux environments using systemd service management.
The patch is one line of code [1]. There is no world in which a reimplementation is cheaper than adding one line of code.
> 3a. This could also have lead to an evaluation of the secure shell protocol being an adequate remoting utilitiy for such environments.
OpenSSH is a standard. Why would you switch to something else when adding one line of code solves the problem?
Your alternatives might be reasonable in a world were everyone has loads of time to go around finding optimal solutions to problems. In reality it's all just volunteers who do this for fun. After the OpenSSH maintainers rejected the patch, no-one else was going to spend more than 5 minutes looking for an alternative for a one line patch.
> My main point of critique about the entirety of decision-making that lead up to this incident is that there is indication that reasonable security practices were neglected in favor of personal ambition,
Honestly, given the choice between the one-liner we used to have and the patch that has now been merged to OpenSSH, the former seems more obviously correct. There was no obvious security issue here, that was never brought up as an issue. I don't really know what was going through the minds of the OpenSSH maintainers, but if >90% of your users are using a patched version you disapprove of, that should trigger some scratching behind the ears.
Even now, most people using OpenSSH on Linux have it patched for socket activation which upstream doesn't like. Yet only actually running the daemon on demand is just obviously superior.
[1] https://git.centos.org/rpms/openssh/blame/SOURCES/openssh...
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 0:21 UTC (Sun) by wahern (subscriber, #37304) [Link]
We can partly tell what was going through the maintainer's mind from the original thread in August 2018, "openssh 7.6 and 7.7 on Oracle Linux 7 (compiled from source) doesn't start correctly with systemd", https://lists.mindrot.org/pipermail/openssh-unix-dev/2018... The most relevent participants in that thread were Damien Miller (maintainer) and Peter Stuge (long-time contributor, IIUC).
The initial one-line approach depending on libsystemd was quickly rejected. It seems like Damien's express reasoning was not wanting a functional dependency, especially one that's also a literal dependency on an LGPL library, but also for other architectural reasons between how OpenSSH behaves and the expectations of systemd.
Peter Stuge also rejected the one-line approach outright because, "At the very least it introduces a dependency on libsystemd into sshd, which is undesirable for reasons of security and convenience."[1] I think Damien would have been of a similar mind regarding security, except in this case his expressed focus seemed to lie elsewhere.
Most relevant messages in that thread:
[1] Peter Stuge's in-depth analysis of systemd integration describing fundamental mismatches between OpenSSH process management requirements and systemd expectations, but also including a patch implementing sd_notify in-tree: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...
[2] Damien Miller agreeing with the analysis, but coming to a different conclusion than Peter re the prudence of any systemd integration: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...
[3] Peter Stuge trying to change Damien's mind: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...
[4] Damien Miller replying to another participant that an optional dependency (literal, functional?) is a dependency like any other: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...
[5] Peter Stuge replying to several other points made else thread: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...
> Even now, most people using OpenSSH on Linux have it patched for socket activation which upstream doesn't like. Yet only actually running the daemon on demand is just obviously superior.
OpenSSH implements very careful and deliberate semantics on reload which don't work well with any of systemd's service types. See Peter's messages above. Fundamentally, and similar to the case of sandboxing, there's a limit to what behaviors and features systemd can provide *outside* the process(es) it starts, even with partial solutions like sd_notify. systemd integration can introduce unnecessary complexity and confusion, if not outright barriers, to implementing the best solution. Usually this won't matter because systemd's options are usually a "step forward" (to quote Peter) in relation to the *average* quality of process management in service daemons. But OpenSSH isn't your average project. Reasonable people can disagree on the prudence of systemd integration, but many of the issues under contention aren't obvious.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 11:21 UTC (Sun) by bluca (subscriber, #118303) [Link]
That is self-evidently not true, given the fact that all distributions have shipped openssh with the notification + socket activation out-of-tree patches for a decade or so now, and they work well. And the notification patch is now upstream, and supports reloading too - I made sure of that.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 9:44 UTC (Mon) by rnestler (subscriber, #160299) [Link]
ArchLinux apparently didn't patch openssh to link to libsystemd/lzma according to https://archlinux.org/news/the-xz-package-has-been-backdo...
> Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command: ldd "$(command -v sshd)"
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 10:24 UTC (Mon) by bluca (subscriber, #118303) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 3:06 UTC (Sun) by tilt12345678 (subscriber, #126336) [Link]
I am under the impression that the "rock and the hard place" really were the OpenSSH and the systemd developers. On the one hand, "this is undesirable" is not a sufficiently scientific explanation for why a project refuses to implement a feature that is required for supporting 90% of its use-cases. On the other hand, a service manager that requires a service software to link to a huge library that does who knows what is a questionable architecture to say the least.
Next time something like this happens, do not hesitate to halt a release over a single line of code. Both sides have made their point. A resolution could not be achieved. Time to escalate. Take the problem upstairs. Community projects also have an "upstairs" - the public. Debate it there, solve it there. This really has nothing to do with fun, release-pressure, professional courtesy or anything. It affects all.
Anyhow, i understand why this was a difficult situation.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 3:13 UTC (Sun) by pizza (subscriber, #46) [Link]
Uh, no. Until "the public" has packaging and/or commit rights, so they are literally incapable of "solving it".
And "the public" will, 99.999% of the time, chose "I don't give a F- about security, I just want it to work, the faster the better."
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 8:08 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]
OpenSSH could have just merged that small patch to manually do the completion notification, without requiring libsystemd.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 14:33 UTC (Sun) by kleptog (subscriber, #1183) [Link]
But a resolution *was* reached: the upstream developers rejected the patch, and all the Linux distributions included it anyway. The systemd developers have nothing to do with this, they created the notification protocol and the library but are not responsible for getting it included into daemons like OpenSSH. That's purely between the OpenSSH developers, the Linux distributions that ship it and their users.
That's the beauty of open-source: there doesn't need to be one solution. Everyone can choose their own solution. Forking a project is not a problem of itself. It's often a solution in difficult situations, like this one.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 17:43 UTC (Mon) by ibukanov (subscriber, #3942) [Link]
Moreover, given that OpenSSH uses the model of first starting to listen for the connection and then forking the systemd notification support can be implemented without touching OpenSSH using other notification mechanisms that systemd supported and that were used prior notify changes.
So refusion of OpenSSH to include the changes were rather reasonable and was strong indication that the changes were problematic.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 18:43 UTC (Mon) by pizza (subscriber, #46) [Link]
Um, no. That (compile-time) change had no effect on distros that don't utilize systemd.
Given that every user of systemd+openssh benefited from that feature (whether or not they recognize or accept it), it had no effect on anyone else.
> Moreover, given that OpenSSH uses the model of first starting to listen for the connection and then forking the systemd notification support can be implemented without touching OpenSSH using other notification mechanisms that systemd supported and that were used prior notify changes.
Um, no. The prior mechanism was not reliable, hence the entire reason for the patch (and its nearly universal use) to begin with.
> So refusion of OpenSSH to include the changes were rather reasonable and was strong indication that the changes were problematic.
Except they've since accepted the change, which is a strong indication that this change was indeed beneficial after all.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 16, 2024 10:04 UTC (Tue) by kleptog (subscriber, #1183) [Link]
But what about reload notification (I've received your signal to reload the config and am done) and stopping notification (I've received your shutdown signal and am on it)?
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 9:25 UTC (Sat) by ballombe (subscriber, #9523) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 13:15 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link]
It's *nice* and *ideal* to have no downstream patches, but when upstream is intransigent it is sometimes essential.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 13:41 UTC (Sat) by bluca (subscriber, #118303) [Link]
Until the next upstream release, which will finally include the functionality natively, to be fair. It only took a world-ending supply chain attack to get that 8-years-old patch merged. Now if we just could get Jia Tan to compromise another dependency of openssh, maybe we could get the socket activation patch merged too :-)
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 15:17 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]
Sounds to me more like a deficiency of systemd than one of OpenSSH ... SCNR! Notwithstanding this, if OpenSSH now has decided that they will perform the integration and take care of it in the log run, Kudos to them. It needs to be done. If i would have to guess, i'd say Linux with systemd is the most frequent environment OpenSSH is run in.
I hope that other such controversies do not require (let me quote another post here) "a world-ending supply-chain attack" to be resolved. Please, everyone, be more cooperative.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 17:45 UTC (Sat) by flussence (subscriber, #85566) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 18:46 UTC (Sat) by Wol (subscriber, #4433) [Link]
As part of its shutdown sequence, it shut down networking. Then it had a habit of hanging.
When it's headless, in a lights-out co-lo, that is an EXPENSIVE problem, unless you can log in to the UPS, kill power to the server, and then send a "wake on wan" command.
Systemd, on the other hand, can be configured to say "this system is going down, and if something hangs it WILL NOT interrupt the shutdown.
Whether you like systemd or not, a *functioning* pid1 is much simpler, smaller, and easier to understand than most (all?) *functional* alternatives. A bare systemd is much bigger than a bare SysVInit, but as soon as you throw in all the service scripts, SysV grows like Topsy. With massive amounts of cargo-culted bash code.
Systemd, if you tell it to go down, IT GOES DOWN.
Cheers,
Wol
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 21:16 UTC (Sat) by flussence (subscriber, #85566) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 21:44 UTC (Sat) by Wol (subscriber, #4433) [Link]
Cheers,
Wol
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 0:00 UTC (Sun) by cesarb (subscriber, #6266) [Link]
Even a system which fails to shut down in a normal office can be a problem.
Story time.
On a former company, for some reason one particular desktop failed to shut down correctly, and when that happened, some stuck networking daemon in it caused a broadcast flood. Since the office network was flat, the broadcast flood also hit the embedded device which controlled the keycard entry to the office, which after some time slowed down and then failed. Since the managers had left the backup device (a remote control keyfob which could also cut power to the magnetic locks) locked within their office, everyone was stuck outside.
As luck would have it, that security system was badly designed (by the former occupier of that office space), and if you knew where to push, you could open a hidden cover on the *outside* of one of the doors, which gave access to the wiring (and the power supply and battery) for that door. A few borrowed tools later, and one of the wires was cut (I chose to cut the red one, because it's tradition, but any of them would have worked), permanently disabling the lock so everyone could get in.
The problem was then properly solved by enabling broadcast storm control in the switch, isolating the keycard control device in a separate "things" VLAN, and convincing the managers that they really need to take that keyfob home with them. But another thing that could have prevented the issue? This was pre-systemd (i.e. upstart) Ubuntu; if it were modern Ubuntu, systemd would have timed out, force killed the stuck daemon, and force shut down the desktop, long before the embedded device could get too overloaded to recover on its own.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 17:51 UTC (Sun) by flussence (subscriber, #85566) [Link]
And — assuming I'm not dealing with a load of hor-^WFUD here, which I'm really starting to think this is — I would really like to know details of how the init system I'm using is supposedly insecure in a dire way for lack of startup micro-optimisations so I can _fix_ it. I'm sure everyone else hacking on non-systemd systems wants to know what the deal is too.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 14:34 UTC (Sun) by jejb (subscriber, #6654) [Link]
This is completely the wrong takeaway: If you imagine a world where openssh had accepted the patch and systemd could be a configured dependency in upstream it would have done nothing to disrupt the attack ... everything would still have happened the way it did. More theoretical scrutiny on libsystemd linked to openssh wouldn't have flagged the xz dependency as a vulnerability.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 15:33 UTC (Sun) by ovitters (guest, #27950) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 18:43 UTC (Sun) by ballombe (subscriber, #9523) [Link]
the attackers could have performed it in a lot of different ways.
From their point of view it must be an abject failure, unless they were trying to make a point.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 14, 2024 20:54 UTC (Sun) by smoogen (subscriber, #97) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 7:38 UTC (Mon) by calumapplepie (subscriber, #143655) [Link]
Perhaps many of those, done correctly, would be more noticeable. Overwriting a critical function in a secure program by linking in a compromise library has a lot more potential to go unnoticed than, say, an extra file in authorized-keys. But if you're looking for it, this can be checked for statically fairly easy; and I wouldn't be surprised if tools popped up to do just that in the near future. Heck, I might try my hand at writing one; a little script over objdump -T, wiring it into debian maintainer tools... etc.
The patch to OpenSSH did not enable the vulnerability, and thus was not flawed. In fact, I understand it has been accepted upstream. Don't rail against patches you don't fully understand.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 10:39 UTC (Mon) by kleptog (subscriber, #1183) [Link]
I agree with you, but just wanted to clarify some points. AIUI the patch included by upstream is not the patch distributions used. Two different patches for the same problem.
The distribution patch was adding a single line of code plus some autoconf magic to configure it. For packagers this is the obvious choice since they don't necessarily understand all the code and it's a minimal "obviously correct" patch.
The patch applied by upstream [1] avoids linking an external library and vendors the sd_notify() function, tweaking it for the OpenSSH code base. Both the distribution patch and a version of this patch were offered to the OpenSSH developers in 2018, and rejected. This patch is much larger and not as "obviously correct" so given a choice, packagers are going to go for the short patch if they get a choice.
[1] https://github.com/openssh/openssh-portable/commit/08f579...
It's interesting to compare this to the Linux kernel. There there is social pressure to not have external patches and to upstream everything. There are all sorts of good reasons for this. But the flip-side is that the kernel developers have to not randomly stand in the way of feature requests just because they don't care about the use-cases. The OpenSSH developers have opinions about the use-cases (e.g. socket activation) but also want to discourage downstream users from patching. You can't have it both ways.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 15:54 UTC (Sat) by DOT (guest, #58786) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 17:00 UTC (Sat) by NYKevin (subscriber, #129325) [Link]
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 13, 2024 18:39 UTC (Sat) by branden (guest, #7029) [Link]
Early in the piece he goes to a familiar well. To his credit, he critiques it.
“Given enough eyeballs, all bugs are shallow.” -- "Linus's Law", glibly popularized by Eric Raymond
But there's a bigger problem. Let's call it the engineering manager's corollary to Linus's Law
"Eyeballs cost money. How are eyeballs going to deliver shareholder value THIS QUARTER? I've got a promotion or lateral move lined up that depends on shipping on time. Are these eyeballs going to get product shipping faster? Then f*** eyeballs."
Silicon Valley, Redmond, the Grand Duchy of Luxembourg (site of SuSE HQ these days)--these places will not solve the problem. They are constitutionally unable to connect what needs to be done to their bottom line.
I predict that Andreas Freund's next performance review at Microsoft will go spectacularly well for him. He has my thanks and congratulations!
After that, if he continues in his recent vein, he's going to have to go to work elsewhere if he desires advancement.
That won't be his fault or even his boss's fault. Tech firms are structurally incapable of incentivizing robust software quality.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 10:48 UTC (Mon) by walex (subscriber, #69836) [Link]
"Tech firms are structurally incapable of incentivizing robust software quality."
That is not really different from any other business: the incentive is always to dump (externalize) costs on others and to defer costs as much as possible into the future. Whether it is toxic pollution of air, water, soil, or pollution of software with security and reliability issues.
The outcome is that we have toi learn two things: #1 survive unreliable (including extensively backdoored) products and processes #2 find some mechanisms to incentivice "good enough" quality. Neither are easy tasks.
As to software backdoors I would expect that nothing has changed from the past: in the past a significant percentage of every important organization have been moles or informers for management or for rivals; if the spy services of the major players are doing their job right I would expect that 5-10% of technical employees at Google, Microsoft, Facebook, NVIDIA, ... and most significant open source projects to be moles or informers. It is both very easy to arrange and very cheap, and much less risky than planting moles and informers into military or government organisations. Not for nothing both the RF and PRC governments have been imposing that any critical hw or sw be entirely designed and manufactured within their countries, and the UK government only accepted Huawei products where each line of sw in them had been vetted by government appointed auditors.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 11:37 UTC (Mon) by paulj (subscriber, #341) [Link]
As a result of this illegal wiretapping, a Greek manager in Vodafone died. Either by suicide or murder.
What we need to take away from the XZ Backdoor (openSUSE News)
Posted Apr 15, 2024 17:03 UTC (Mon) by hkario (subscriber, #94864) [Link]
And the solution is the same as with all those other industries. Is your widget safety critical? Then it needs to follow verified 3rd party certified testing. That's done through legislation.
That's also the reason why Americans don't like "legislation" as a solution: they have been under 40 years of constant propaganda telling them how "the gubbement" doesn't work, paid for by the very companies that want to make stuff as cheaply as possible because next quarter financial results are the only thing that matters. Boeing is the poster child of that approach.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK