How Golang Manages Its Security Supply Chain

Feature image of the Enigma Machine by Florian Pircher from Pixabay.

The success of open source software also brings new security concerns. That was one of the most interesting takeaways from a recent keynote talk by Russ Cox, the tech lead for the Go programming language.

Also a distinguished engineer at Google, Cox warned that “We are all struggling with a massive shift that has happened in the last 10 or 20 years in the software industry. For decades software reuse was only a lofty goal. Now it’s very real.”

And by the end of the lecture, Cox had brought his warning home. “The programming environments for languages like Go, npm, and Rust make it trivial to download and run source code published by strangers on the internet. We all do it, and no one is checking the code until there is a problem.”

But Cox wasn’t just delivering a warning. There’s also a hopeful message — that new tools, services, and infrastructure will help meet those challenges.

And Cox demonstrated this by explaining the many supply chain-protecting steps that Google takes to ensure the security of Go.

Danger from Dependencies?

Cox delivered his keynote at November’s ACM SCORED (the group’s official workshop on Software Supply Chain Offensive Research and Ecosystem Defenses, coinciding for the last two years with their annual conference on Computer and Communications Security). To drive home the point, Cox showed his audience the dependency graph for Kubernetes as plotted by Google’s Open Source Insights project.

“There are hundreds of dots here,” Cox says, “each one abstracting away some collection of relevant machines… Each one of these is a different potential attack site, not to mention a potential source of vulnerabilities.

“We should be concerned with who has access to each of these projects, who might have access in the future, what infrastructure they use, and so on. For the most part, we don’t have visibility into any of that — so we ignore it. But it’s all still there.”

Later Cox shared the final tally: “Kubernetes uses 3,000 packages from 200 dependency modules…”

Cox would like to see the data converted to machine-readable formats for easier (and more systematic) analysis. But “for now I think we understand certain parts well enough to identify important weaknesses and work on strengthening those.”

One Key Is Keys

But what’s the next step? Once you understand your software’s supply chain, how exactly do you strengthen it? The most effective measure, Cox says, is cryptographic signatures, “which make it impossible to nefariously alter code between signing and verifying.

“This completely removes download servers, proxies, and other network middleboxes as potential attack sites.”

This does require confirming keys, but to simplify that process, there’s the Go checksum database, which Cox describes as “protecting millions of Go developers,” by housing the SHA-256 checksum for every version of every public Go package.” It comes into play when users execute the “Go” command (which automates the downloading and installing of Go packages) — since the public key for every package is already hard-coded into the Go command’s source code, and it’s ready to query the database for the expected checksum.

Capslock and Fuzzing

This year Google released another security-checking tool called CapsLock which aims to bring some transparency to what packages are capable of. “The vision is that you can use this tool to decide whether you want to take on a new dependency,” Cox told his audience, “and also to watch that existing dependencies don’t start doing new things when you upgrade them.”

If something like Capslock had existed for Java it would have flagged that Log4j could download and execute code for remote servers — and maybe fewer people would’ve used Log4j, or that feature might’ve been removed sooner.

Google also started (and funds) OSS-Fuzz, a program providing continuous “fuzzing” (the random inputs that help software in screening for common vulnerabilities like buffer overflows). “As of August 2023, OSS-Fuzz has helped identify and fix over 10,000 vulnerabilities and 36,000 bugs across 1,000 projects,” according to its README page on GitHub.

And for operating systems, Google also runs a separate service called Syzkaller, which has found nearly 5,000 bugs in the Linux kernel (as well as hundreds of still-unaddressed open issues).

Cox warns that “The effectiveness of fuzzing at finding vulnerabilities in modern software is a serious indictment of modern software practices… We can’t keep playing whack-a-mole with buffer overflow and use-after-free bugs. We have to find systematic ways to do better.

“Specifically we need to stop using C and C++ and move to memory-safe languages.”

SBOMs and Vulnerability Databases

But what may be just as important is actively monitoring the supply chain for the presence of any newly identified weaknesses. Software Bill of Materials can help — a list of all the components (including third-party dependencies), as well as their version numbers. And while it’s never been precisely defined what should go into a Software Bill of Material (SBOM), every Go binary now includes a list of every Go package that contributed to its source code. “There’s no build option to omit this information. or remove it after the fact — even though people have asked for that. You can rely on it being present in any compiled Go binary.”

Obviously, you’d want to check these components against a database of known vulnerabilities — and for that, there’s the Go vulnerability database. Curated and reviewed by the Go security team, it tracks which versions of packages contained the vulnerability.

There’s also the larger Open Source Vulnerability database, supplying the same package and version information for known vulnerabilities. This database worked with the Go security team to define a clear format for describing vulnerabilities (According to its FAQ, the OSV format “enforces version specification that precisely matches naming and versioning schemes used in actual open source package ecosystems,” while also being easily understandable by both humans and automated systems.)

Cox tells the audience that OSV is now the “native” vulnerability-recording format for not just Go, but also Rust, Python, and the GitHub Security Advisory database, among others.

Finally, Cox turned to Govulncheck, a command-line tool released in July that can analyze both binaries and codebases for vulnerabilities (checking against the Go vulnerability database).

To eliminate false positives, it even checks to see which functions are being called from packages (since some functions in a vulnerable package may not be affected by its vulnerability). Checking the source code allows an even more sophisticated analysis — which can even identify false positives that are sometimes flagged by other tools.

The trick is to make this vulnerability checking easily accessible and widely available. “Scanning isn’t worth anything if no one uses it,” Cox adds, “so we’ve also integrated vulnerability scanning into the IDE experience.” He put up a screenshot showing VSCode offering Govulncheck as a choice in the dropdown menu, to address a problem dependency that’s already been highlighted in the code-editing window with a squiggly line. “The earlier in the development process we can provide this information the easier it is for users to address these problems.”

And in addition, Govulncheck is also integrated into Google Cloud’s Artifact Analysis API, so it can scan for vulnerabilities in already-running code in production.

So while there are concerns about software supply-chain security, “It feels like change is coming,” Cox told his audience. “I hope that we will look back 10 years from now and marvel at how much better things are.

“And I hope you’ll all help.”

WebReduce

• Drew DeVault was interviewed about “Hare,” his 20-month-old attempt at a “100-year programming language.”
• Cory Doctorow discusses his (released in September) book, The Internet Con: How to Seize the Means of Computation (from digital spaces to infrastructure).
• Can robots replace Michelangelo? The quest to perfect sculpting by robot.

David Cassel is a proud resident of the San Francisco Bay Area, where he's been covering technology news for more than two decades. Over the years his articles have appeared everywhere from CNN, MSNBC, and the Wall Street Journal Interactive...

Read more from David Cassel