Make no mistake about it. The Apache Log4j’s Log4Shell security hole was an all-time awful code disaster. I mean, by National Vulnerability Database (NVD) count, it had a 10.0 CVSSv3 score. That’s on a 0.1 to 10 scale.

As security company Nextron Systems’ head of research, Florian Roth tweeted, “The #Log4Shell vulnerability isn’t just an RCE [Remote Code Execution] 0day. It’s a vulnerability that causes hundreds and thousands of 0days in all kinds of software products. It’s a 0day cluster bomb.”

But, hey, it was patched months ago. True, there was a security hiccup with the first Log4j patch, but it was quickly fixed. So, we’re all good now, right? Right!?

Wrong.

Exploits Persist

As the US Cybersecurity and Infrastructure Security Agency (CISA) and the US Coast Guard Cyber Command (CGCYBER) recently reported in a joint Cybersecurity Advisory (CSA) notification, hackers, including state-sponsored advanced persistent threat (APT) actors, are still exploiting CVE-2021-44228  (Log4Shell) in VMware Horizon and Unified Access Gateway (UAG).

But guess what? There’s more!

The US Department of Homeland Security (DHS) Cyber Safety Review Board’s (CSRB) report on Log4j, found “The Log4j event is not over. Log4j remains deeply embedded in systems, and even within the short period available for our review, community stakeholders have identified new compromises, new threat actors, and new learnings.”

The attacks just keep coming and coming. As Chad Skipper, VMware’s Global Security Technologist, said, “Since January 2022, VMware NSX Network Detection and Response has tracked over 25 million exploit attempts against Log4j.” And there are far, far more organizations out there that don’t have any security system protecting and tracking their problems.

Log4j Is Everywhere

Why is Log4j such a persistent pain in the rump? First, it’s a very popular, open source Java-based logging framework. So it’s been embedded into thousands of other software packages. That’s no typo. Log4j is in thousands of programs. Adding insult to injury, Log4j is often deeply embedded in code and hidden from view due to being called in by indirect dependencies.

So, the CSRB stated that “Defenders faced a particularly challenging situation; the vulnerability impacted virtually every networked organization, and the severity of the threat required fast action.” Making matters worse, according to CSRB, “There is no comprehensive ‘customer list’ for Log4j or even a list of where it is integrated as a subsystem.”

Welcome to how we use open source code. That’s a big reason Software Bills of Materials (SBOM) have become essential for securing the software supply chain.

So it was that “Enterprises and vendors scrambled to discover where they used Log4j. The pace, pressure, and publicity compounded the defensive challenges: security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and ‘patching fatigue’; defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors, and responders found it difficult to find authoritative sources of information on how to address the issues,” the CSRB said.

The best responders from the open source community collaborated while working through weekends and the December holidays. They were largely successful in finding and fixing the bugs. However, the CSRB claims that “few organizations were able to execute this kind of response” as quickly or efficiently as needed. Part of the problem was deploying the patches was “also a risk decision, forcing a tradeoff between possible operational disruption and timeliness, completeness, and compensating controls.”

What You Can Do

So, what can we do? The CSRB has many good suggestions on how to protect yourself. Most of them, you already — or at least you should — know.

But if you haven’t been doing them yet — and I know some of you haven’t — it’s high time you did. After all, as the CSRB Board states, Log4j is an “endemic vulnerability” and “vulnerable instances of Log4j will remain in systems for many years to come.”

My final word, for now, comes from Google’s VP of infrastructure, Eric Brewer.  “If you are using open source, you can’t expect other people to magically fix security issues for you,” If you use open source code — and who doesn’t? — you, too, must help maintain and patch its problems. We are all in this together.

The New Stack is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Coast.

Featured image via Unsplash.