A few relevant quotes
source link: https://lwn.net/Articles/967420/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
A few relevant quotes
I'm on a holiday and only happened to look at my emails and it seems to be a major mess.— Lasse Collin
The reality that we are struggling with is that the free software infrastructure on which much of computing runs is massively and painfully underfunded by society as a whole, and is almost entirely dependent on random people maintaining things in their free time because they find it fun, many of whom are close to burnout. This is, in many ways, the true root cause of this entire event.
Incredible work from Andres. The attackers made a serious strategic mistake: they made PostgreSQL slightly slower.
There is no way to discuss this in public without turning a single malicious entity into 10 000 malicious entities once the information is widely known.Making sure the impact and mitigations are known before posting this publicly so that everyone knows what to do before the 10 000 malicious entities start attacking is just common sense.
Again the FOSS world has proven to be vigilant and proactive in finding bugs and backdoors, IMHO. The level of transparency is stellar, especially compared to proprietary software companies. What the FOSS world has accomplished in 24 hours after detection of the backdoor code in #xz deserves a moment of humbleness. Instead we have flamewars and armchair experts shouting that we must change everything NOW. Which would introduce even more risks. Progress is made iteratively. Learn, adapt, repeat.— Jan Wildeboer
(Log in to post comments)
A few relevant quotes
Posted Mar 30, 2024 17:24 UTC (Sat) by jcdickinson (subscriber, #168509) [Link]
A few relevant quotes
Posted Mar 30, 2024 22:24 UTC (Sat) by draco (subscriber, #1792) [Link]
Based on the evidence as reconstructed so far, I believe Lasse is a victim that deserves the benefit of the doubt
A few relevant quotes
Posted Mar 30, 2024 23:58 UTC (Sat) by MarcB (subscriber, #101804) [Link]
This is made worse by the fact that he made some benign contributions to various projects, likely to earn trust and reputation, and in doing so interacted with several people.
Admittedly, he also seems to have made more questionable contributions, like removing a filter for escape sequences contained in file names: https://github.com/libarchive/libarchive/pull/1609
Also, another malicious one to xz-utils: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da... (Fix: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05...)
All in all, this is extremely damaging. Suddenly people who accepted a benign - or questionable at worst - commit are suspected as co-conspirators.
A few relevant quotes
Posted Mar 31, 2024 0:46 UTC (Sun) by dullfire (subscriber, #111432) [Link]
Considering how much of a long-term game appears to have been in play here (at least from what I have heard), name sound entirely questionable. Unless there's people reliable people who've met the person, there is no reason to believe that is their real name, or has any reflection on their real identity.
And grouping people together that way (especially with a complete lack of *any other evidence*) is almost never helpful.
A few relevant quotes
Posted Mar 31, 2024 1:03 UTC (Sun) by lambda (subscriber, #40735) [Link]
There is no good reason to suspect that any of these have any relationship to the attacker's actual identity.
A few relevant quotes
Posted Mar 31, 2024 13:15 UTC (Sun) by nix (subscriber, #2304) [Link]
A few relevant quotes
Posted Mar 30, 2024 18:29 UTC (Sat) by gmgod (subscriber, #143864) [Link]
A few relevant quotes
Posted Mar 30, 2024 20:39 UTC (Sat) by geuder (subscriber, #62854) [Link]
However, the practice to use release tar balls which do not match what is under version control made the wrongdoing more difficult to detect. And I can see no single valid reason to do so, it should be extremely easy to stop.
A few relevant quotes
Posted Mar 30, 2024 22:33 UTC (Sat) by ballombe (subscriber, #9523) [Link]
You cannot usually redistribute the whole git repository and a git archive of a commit
is not reproducible.
It is more important to protect against webhost compromise than upstream going rogue.
Most of the payload was checked in the GIT repository anyway and nobody noticed.
A few relevant quotes
Posted Mar 31, 2024 7:44 UTC (Sun) by geuder (subscriber, #62854) [Link]
Microsoft uses codesigning, but I assume few in this forum would support the idea that that is a sufficient measure to trust software. Being able to inspect the code in a favorable form with the support of version management tools is an essential part of FOSS. Having a big amount of m4 stuff, illegible to most of us in any reasonable amount of time is not better than distributing binaries.
> You cannot usually redistribute the whole git repository
Why not? Are you referring to size?
My Yocto builds use BB_GIT_SHALLOW [1], so I clone only once until the next update. Yes, the tooling would need improvement, so it could handle deltas.
> git archive of a commit is not reproducible.
Poor tooling once again.
> Most of the payload was checked in the GIT repository anyway and nobody noticed.
So we need to learn to treat test data as untrusted, like you treat inputs from the internet. I guess it should be possible to use landlock not to read from test data locations during the build phase, before entering test phase. And the test phase not being allowed to change the binaries produced anymore. (Guessing, haven't used landlock myself yet.)
> It is more important to protect against webhost compromise than upstream going rogue.
Not sure I am following here. I think most build systems verify a hash already. The bigger risk is really do I have a hash belonging to good code in the beginning.
My point with not 100% agreeing with the original quote that FOSS did great in this case is: Yes, the aftermath is handled much better than any closed source company would do. But our tools and processes have many weaknesses, too. We tend to trust FOSS developers because they are good guys. Many IT managers trust Microsoft because they are such a successful company. And each side is convinced the other side is just naive.
[1] https://docs.yoctoproject.org/bitbake/dev/bitbake-user-ma...
A few relevant quotes
Posted Mar 31, 2024 12:40 UTC (Sun) by ballombe (subscriber, #9523) [Link]
> Why not? Are you referring to size?
Yes. It would not be reasonable for debian to ship the whole git archive as orig.tar.gz
>> It is more important to protect against webhost compromise than upstream going rogue.
>Not sure I am following here.
The git repository can have been tampered with.
> I think most build systems verify a hash already. The bigger risk is really do I have a hash belonging to good code in the beginning.
Yes, we still need code signing.
A few relevant quotes
Posted Mar 31, 2024 16:59 UTC (Sun) by geuder (subscriber, #62854) [Link]
A bare shallow clone should not be significant different in size and you can immediately see whether your SHA-1 matches or not.
> The git repository can have been tampered with.
Tampered such that the SHA-1 of your release has changed? Yes that's possible, but you should know what is a good SHA-1 to base your build on. Either from a signed commit/tag or a signed release note.
Or are you referring to SHA-1 collisions? Well, they might come some day, but at the moment I am not overly worried yet. To hide some payload it should not be done in the tip, but a bit deeper in the history. That seems to be many years out.
A few relevant quotes
Posted Mar 31, 2024 8:47 UTC (Sun) by walters (subscriber, #7396) [Link]
A few relevant quotes
Posted Mar 31, 2024 12:27 UTC (Sun) by geuder (subscriber, #62854) [Link]
"make dist" is the step I have never liked, mostly out of debugging/reproducibility reasons. As I wrote in a sibling it's close to distributing binaries. And in the current case it was a relevant part of the exploit.
About SHA-1 I personally haven't been too worried yet. Creating git hash collisions hasn't been demonstrated yet. But maybe inserting "suitable" test data will allow that some day.
A few relevant quotes
Posted Mar 31, 2024 6:11 UTC (Sun) by cozzyd (subscriber, #110972) [Link]
A few relevant quotes
Posted Mar 31, 2024 16:08 UTC (Sun) by dullfire (subscriber, #111432) [Link]
A few relevant quotes
Posted Mar 30, 2024 23:21 UTC (Sat) by MarcB (subscriber, #101804) [Link]
Nothing was "proactive". Basically all rolling release distributions happily shipped it; Gentoo and Arch were just lucky the attackers didn't consider them relevant enough and the backdoor code wasn't even injected in their build environments. I had the package on my Arch system, but the backdoor was indeed not present.
Archlinux also would have been unaffected, since they neither carry the libsystemd patch nor SELinux. For Gentoo it likely would have depended on USE flags.
Only a single person (Andres Freund) was vigilant, and even he says that a lot of lucky coincidence was involved (and this wasn't even on one of the rolling release distributions). Maybe it would have been discovered by someone else before the Fedora 40 and Ubuntu 24.04 releases in April, maybe not.
Also, the statement on Mastodon "The backdoor code was inserted only under very specific circumstances in the build process" is IMHO very misleading. The "very specific circumstances" were: Building a package for any of the largest distributions, for the most common architecture. Ignoring the mobile and embedded world, this would have easily affected 80% of all Linux installations. Likely more.
The statement from Russ Allbery is (unfortunately) much closer to 100%. It is pretty clear to me, that they specifically targeted a quasi-core project with just a single maintainer who was already struggling significantly and publicly. This is as malicious as it is smart. A truly proactive FOSS world would have acted on the maintainer's situation: Either by supporting him or by removing the package from core components (ideally both).
A few relevant quotes
Posted Mar 31, 2024 0:11 UTC (Sun) by judas_iscariote (guest, #47386) [Link]
You can't. distributions are built with xz compressed packages, rpm package payloads specifically.
A few relevant quotes
Posted Mar 31, 2024 2:17 UTC (Sun) by salimma (subscriber, #34460) [Link]
A few relevant quotes
Posted Mar 31, 2024 2:54 UTC (Sun) by draco (subscriber, #1792) [Link]
$ rpm -ql zstd |grep lib |grep -vF .build-id
$ rpm -q --provides zstd |grep lib
$ rpm -q --requires zstd |grep ^lib |grep -vE 'lib(c|gcc|stdc)'
liblz4.so.1()(64bit)
liblzma.so.5()(64bit)
liblzma.so.5(XZ_5.0)(64bit)
libz.so.1()(64bit)
A few relevant quotes
Posted Mar 31, 2024 7:18 UTC (Sun) by pbonzini (subscriber, #60935) [Link]
A few relevant quotes
Posted Mar 30, 2024 18:34 UTC (Sat) by engla (guest, #47454) [Link]
Which suggests to me they are treating their build infrastructure as compromised, which is not strange given how their tools including dpkg link to the compromised library (in newer versions).
A few relevant quotes
Posted Mar 30, 2024 18:41 UTC (Sat) by rra (subscriber, #99804) [Link]
A few relevant quotes
Posted Mar 30, 2024 19:32 UTC (Sat) by cjwatson (subscriber, #7322) [Link]
A few relevant quotes
Posted Mar 30, 2024 20:28 UTC (Sat) by birdie (guest, #114905) [Link]
Not mentioning RedHat/IBM who have been heavily investing and sponsoring Linux for almost three decades now.
A few relevant quotes
Posted Mar 30, 2024 21:23 UTC (Sat) by Phantom_Hoover (subscriber, #167627) [Link]
A few relevant quotes
Posted Mar 30, 2024 23:48 UTC (Sat) by tomsi (subscriber, #2306) [Link]
So that no company or organization stepped in to htlp him?
A few relevant quotes
Posted Mar 31, 2024 13:19 UTC (Sun) by nix (subscriber, #2304) [Link]
A few relevant quotes
Posted Mar 31, 2024 3:11 UTC (Sun) by rra (subscriber, #99804) [Link]
If you want to force me to do something in a specific way, you will have to pay me, because at that point it is no longer volunteer and it is no longer fun. And I say this despite the fact that I probably agree with you on 95% of the higher standards that you would like to impose and I am trying to hold myself to them voluntarily. But as soon as you bring legal force to bear, I'm out; that will be the last free software you see from me on my own time. If you want to control me and tell me what to do, that is the definition of a job and I expect a salary that is worth the headaches of having to follow your rules.
A few relevant quotes
Posted Mar 31, 2024 4:21 UTC (Sun) by Phantom_Hoover (subscriber, #167627) [Link]
A few relevant quotes
Posted Mar 31, 2024 5:03 UTC (Sun) by rra (subscriber, #99804) [Link]
I think the trick is to make the obnoxious bits people don't want to do a paid job, while letting the fun coding work that I kind of want to do anyway something for which I might get a bit of a stipend that's less than what's paid for the obnoxious bits but enough to give people some options other than a full-time day job. Those two things may combine in one person in some cases, and not in others. For example, if there was money for free software tied to specific projects that are already being used, I think a lot of maintainers would organize into non-profit umbrella organizations and pool their resources, and that umbrella organization could hire people to do the tedious bureaucratic work that the maintainers don't want to do.
Down that path you potentially get non-profits with a predictable funding stream whose whole reason for existence is to make free software and give it away for free, at a high quality. Which has been the holy grail of the free software movement since day one. It would be like the FSF or the Linux Foundation, except with enough separate funding streams that you could form your own or throw in with other like-minded people if those organizations gave you hives for whatever reason. (And not have to then spend the rest of your life begging other people for money.)
I think you could fund an amazing ecosystem through taxes on just the FAANG companies, if we actually taxed them the way that they should be taxed in any fair social system.
A few relevant quotes
Posted Mar 31, 2024 6:03 UTC (Sun) by cpanceac (guest, #80967) [Link]
Then, most if not all of these big companies are members of the Linux Foundation, so how come this foundation is not funding and supporting these projects as suggested above?
A few relevant quotes
Posted Mar 31, 2024 6:27 UTC (Sun) by rra (subscriber, #99804) [Link]
Corporations do not behave precisely according to this model for a few different reasons. One major one is that they're made up of people and those people have normal non-sociopathic human reactions to the rest of society and often can convince corporations to spend some money on things that are not purely profit-maximizing. But those deviations are bounded aberrations and can't be relied upon.
For a functional society, one either has to design laws to not create legal sociopaths, or one has to constrain those legal sociopaths by forcing them to not externalize their costs and free-ride on the rest of society. The former is extremely hard. We've largely chosen to do the latter. The normal mechanism to do that is taxes, which is how all the other infrastructure that corporations rely on (roads, bridges, water, sewer, etc.) is funded. Free software is not treated as infrastructure.
A few relevant quotes
Posted Mar 31, 2024 11:55 UTC (Sun) by bluca (subscriber, #118303) [Link]
The German-govt backed STF is exploring grants/fellowships for OSS maintainers. We should tax the mega corporations and ring-fence the revenue to fund such efforts.
A few relevant quotes
Posted Mar 31, 2024 15:00 UTC (Sun) by mcatanzaro (subscriber, #93033) [Link]
A few relevant quotes
Posted Mar 30, 2024 22:13 UTC (Sat) by jg71 (guest, #67102) [Link]
I still look after a flock of dedicated servers running various flavours of Slackware in my spare time (because I like doing that), and I am so glad this major mess was stopped as soon as it was.
And I can relate to Russ Allbery's quote. That's why I wrote this comment.
I earn my living with a job that is far removed from open source software.
There's a time of fun being involved when it comes to open source software, and then sneakily real life takes its toll. It's hard to pull out when you put your heart and soul into it, lines are often blurred.
I stepped back 'a little' last year when I realized that it felt like a burden more often than not, maintaining buildscripts mostly, helping out in a local users group, ... you catch my drift.
Life is simpler, yes, but I look back to those achievements fondly.
One needs to reflect upon that from time to time, I think, and draw conclusions.
Nobody talks about that early on. Real life is dynamic, and false priorities really can do harm.
A few relevant quotes
Posted Mar 30, 2024 23:00 UTC (Sat) by ejr (subscriber, #51652) [Link]
Software supply chain security is an ongoing area of research and development. Ain't solved yet, and no simple hammer is going to solve it other than just being offline.
Yeah, I have to use AT&T, but this isn't related to their other thing.
A few relevant quotes
Posted Mar 31, 2024 1:11 UTC (Sun) by carlosrodfern (subscriber, #166486) [Link]
Compare that with the attack to the closed software SolarWinds, which actually made it to the customers.
Anyways, a lot of room to improve still.
A few relevant quotes
Posted Mar 31, 2024 3:06 UTC (Sun) by rra (subscriber, #99804) [Link]
Admittedly, I may have also been muttering a rant about how no one writes sufficient documentation. (In this case, that rant was wrong, though; the behavior I was seeing was sufficiently documented, I was just reading the wrong documentation because I had an erroneous understanding of the problem.)
A few relevant quotes
Posted Mar 31, 2024 6:01 UTC (Sun) by mjg59 (subscriber, #23239) [Link]
I initially spent time developing reverse engineering skills for hobby purposes, but these days a fair amount of it ends up being to figure out how proprietary software is behaving and how to work around it. All else being equal I'd definitely prefer to have the source code, but given a decent incentive it's really not as hard to deal with this in the proprietary world as you might think.
Reproducible builds
Posted Mar 31, 2024 17:11 UTC (Sun) by matkon (subscriber, #109282) [Link]
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK