Open Source Security Process Wishlist

December 29th, 2021

Open Source software (OSS) runs the internet (obligatory xkcd). And it is wonderful. But every now and then, it breaks. And then it may set the internet on fire. OSS is, in this way, both not at all and still really quite different from other software: it ought to be owned, managed, and operated in a responsible manner. And that aspect can often be improved.

Without suggesting that OSS maintainers owe their users any more of their time and effort than they already provide -- this level of support easily warrants a stately support contract -- there are a few methods and practices that can help a software project better manage security incidents and vulnerabilities. The following is a short list of such processes.

Most of this may seem obvious to people in the OSS world, in software engineering, or dealing with software security in more general terms. Some of this is tedious and definitely requires additional cycles, work, and effort. But all of this is, I believe, helpful to your users.

1. Have an obvious contact address (e.g., security@) prominently displayed on your main project's website.

Try to make it as easy as possible for people to contact you without jumping through hoops. Email works really well for this. By making it clear that the contact address is intended for security issues only, you can separate out any other user feedback.

You may wish to consider adding a /.well-known/security.txt file in your web server's document root.

1.1 Provide a public PGP key for that address.

For some security issues, you want to receive information in encrypted form. Yes, everybody hates PGP. But there isn't any realistic alternative that's as widely used.

1.2 Be able to decrypt messages encrypted for that key.

Having a public key posted doesn't help you much if you lost the private key or don't know how to use PGP. Now this quickly turns into additional requirements, such as managing a shared private key across the team managing incoming security reports or other security aspects of the project, including key rotation and all the other fun bits here. I'm sorry.

1.3 Actually read the incoming mail.

A lot of incoming mail to this publicly posted address will be spam, noise, and off-topic emails. Set up a rotation amongst people on the security team to monitor this inbox.

I'll leave out all the other details surrounding handling of disclosure coordination, dealing with extortion, bug bounty fame seeking hype requests here. Again, I'm sorry.

2. Have a security-announcements-only mailing list.

It's important that you separate security announcements from any other announcements of informational messages to your users. Some users will only want to hear from you about security releases, vulnerabilities, etc. This facilitates filtering and paying attention.

2.1 Sign announcements to this list using the key from above.

True, hardly anybody will validate the signatures, but you've got a PGP key for your security team, might as well use it. This further helps the community validate and prioritize information from your project.

2.2 Send copies of relevant announcements to e.g., oss-security etc.

Despite having a security-announcements-only mailing list, you still want to ensure that notifications of vulnerabilities or patched releases are widely noticed. The oss-security mailing list is a useful vector to disclose publicly known vulnerabilities; if you qualify, make sure to participate in and disclose to vendors as well. This helps downstream distributors of your software to get notified and take action.

2.3 Link to announcements from official org Twitter account.

If you have social media accounts for your project, make sure to send a link to the announcement email in your mailing list archive from these accounts as well. Twitter, in particular, is widely used for breaking security news, so it may be a good idea to share this information there. By using an official account (instead of relying on individual contributors), you can ensure that your users have an assurance of authenticity of the information.

3. Have a ticket tracking system.

You need to track software changes, so having a proper ticket tracking system can really help you manage security incidents. Within OSS, however, there's an important consideration:

3.1 Allow for confidential tickets.

In Open Source, we like to have all information public. However, with security vulnerabilities, it may be necessary to at times keep some information private or embargoed for a certain amount of time.

3.2 Allow for making confidential tickets public.

But you do eventually want to make all the information public, so make sure that your ticket system can do that. This way, you can have a full discussion under embargo in the ticket, building a full audit trail, and then publish it after release.

3.3 Treat tickets as append-only; don't edit comments.

Your ticket system becomes an audit-trail of the vulnerability discovery, remediation, and disclosure. Resist the temptation to update comments, to add information that is discovered at a later point in time, etc. It is best for everybody's understanding to be able to see how decisions were made, what information was available at what time.

3.4 Let people open and comment on tickets without requiring Yet Another Account.

Similar to providing an easy to find email address, make it easy for your users to contribute in your tickets. Yes, that will invite some level of spam that needs to be tended to, but believe me when I tell you that many security researchers will rather turn away and gripe about your project on Twitter than to create Yet Another Account with your system just to let you know about an issue.

4. Have a revision control system.

Yep, another no-brainer. But as your project takes off, it might be the case that you do not already have a revision control system in place. And do note: your revision control system does not have to be git, and even if it is git, it does not have to be hosted on GitHub.

One important aspect here mirrors the use of a ticket tracking system:

4.1 Be able to have private branches or to later mirror embargoed changes into the public view.

You need to be able to work with a private code base but still retain all the benefits of revision control. You want to keep the commit history, messages, and related discussions in private, and then be able to make them public at a later point.

Without this ability, you might end up "hiding" security fixes in unrelated public commits or with misleading or imprecise commit messages, or be tempted to later manipulate those. This will make it terribly difficult if not altogether impossible for outsiders to understand the changes, or for you to reconstruct them later.

4.2 Ensure security-relevant commits are identified as such, cross reference tracking ticket or CVE.

Your code repository provides a historical record of your changes. What was changed, and why things were changed. Be clear about the importance of the changes and ensure your users can find any related information easily.

4.3 Don't mix security-related and other changes.

This is simple code commit hygiene: separate changes that don't belong together. Make it easy for your users to reconstruct the vulnerable and fixed states, to generate a patch from the diffs, if needed.

5. Auto-comment tickets via commit messages.

Cross-referencing information is the name of the game. This is what incident responders do when they analyze a vulnerability and the (purported) fixes. By having your revision control and your ticketing system talk to one another, you can make this an order of magnitude easier.

5.1 Append commit messages.

Any commit message containing a ticket number should be automatically appended to the ticket in question as a comment. This can easily be done with a backend script that posts or emails the commit information to the ticketing system.

5.2 Append the full code diff.

This may seem like duplication of information (given the next item), but having everything in place ensures that people can read through the ticket and code changes easily side by side and with context.

5.3 Link to the changes in repository.

Your code changes should be browsable in your repository, so add a link to the changeset to the ticket. It doesn't cost you anything and allows people reading through the ticket to jump into the code and move on from there, if they so please.

6. Have a changelog both in your release as well as on your website.

If I fetch your software, I don't want to have to go to a website to see what has changed; if I'm on your website, I don't want to have to download a tarball to extract to see what changed.

Make sure that your changelog is complete and append-only. If you feel like it gets too long, at the very least retain a full-history archive file alongside it. It's plain text, it doesn't cost you anything, and it makes it really easy to later grep through historical releases.

6.1 Separate security changes from others.

Clearly mark changes that are security relevant and don't mix them with other updates. No "we added new feature X, changed behavior of Y, oh, also, we fixed an RCE, and then switched off flag Z and finally added three new translations of our manual pages", please.

6.2 Note any CVE, affected versions, and a rough disclosure timeline.

In your changelog and security announcement, make sure to identify the vulnerability by its CVE identifier. If you were so lucky as to be granted a branded vulnerability complete with MS Paint Logo, sure, mention those names, too.

Clearly identify which versions were vulnerable and roughly when the vulnerability was discovered, reported, fixed, and announced.

6.3 Link security changes to tracking tickets.

Together with your CVE, a full link to the tracking ticket helps people reading the announcement find out all the information, since, as per the above, your tracking ticket does contain all relevant information in one convenient place.

6.4 Briefly explain the possible impact, attack vector.

Don't assume that all of your users understand your product and all it's possible features and configuration options 100%. Likewise, don't assume that all users understand all the information security terms included in the CVE details. Instead, spell it out for them in reasonably simple terms. Include a rough assessment of how difficult the attack is to perform, or whether it has been observed in the wild. This helps your users understand the impact and prioritize their work.

7. Publish signed releases.

Hey, you got that PGP key, make use of it! This helps your users and downstream packagers validate and authenticate your release.

7.1 Publish checksums of releases.

Sure, 99% of your users won't bother, but it really doesn't cost you anything and can help people verify that they got the changeset or release version they wanted.

As a logical follow-up: don't make changes to your release after you have published it. Instead, bump the release version if you have to make changes after publication!

7.2 Don't mix features or other changes with security fixes - point or tiny release security fixes only instead.

If at all possible, try to keep a clear security-features-only release branch of your product. This once more makes it so much easier for your users to identify what changes they get, and lets them apply the updates without worrying about breaking things.

7.3 Backport security fixes to all supported releases.

If you support multiple versions, make sure to backport security fixes to all supported releases. You may be able to get away with prioritizing your mainline product, but releasing fixes for all supported versions at the same time again makes it easier for your users and takes the guess-work out of their impact analysis.

8. Have a well-documented process for handling vulnerability reports.

All of the above means nothing if your security team can't remember what process to follow. Even if you're a one-person maintainer (I see you!), documenting your process will help you remain consistent especially when under tremendous pressure.

8.1 Triage incoming reports.

As noted above, this includes monitoring the security contact address, ticketing system, and likely user mailing lists etc. Try to have a rotation of people attending to and answering incoming requests.

8.2 Open (private) tickets.

Once you've identified and confirmed that a given issue requires attention, open a ticket (embargoed / private if needed). Begin collecting information in the ticket as discussed above.

8.3 Branch code changes.

Cut a new branch for the given vulnerability. If you already have a CVE, tag it as such. Ensure all code changes go into this branch.

8.4 Validate & review.

Verify, confirm, circle back with the original reporter(s), check with your SMEs, update your tickets, and rinse and repeat.

8.5 Tag the release.

Once you know what you're releasing, tag it. Again, make sure not to pollute the security fix patches with irrelevant or unrelated changes. Tag what you will release, and if you did release and need additional changes, create a new tag!

8.6 Publish the release, all code changes, your security ticket, and announce the vulnerability and fix.

The order here doesn't matter too much, but make sure this happens more or less at the same time. Open your previously private ticket(s), merge the private code changes into the public repository (while retaining the revision history!), and push the announcements to your website, your security-announce-only mailing list, and then share it elsewhere on the internet.

As I said, a lot of this is really tedious and a lot of work, virtually all of it thankless (but you already knew this, because you are in Open Source). Many OSS projects do most of the above, but sticking to it all with routine is the key part. That allows you to build a reliable timeline after the fact, perhaps even years later.

Don't let online panic derail you from that, and don't take shortcuts.

To all OSS maintainers: thanks for all your work!

December 29th, 2021

P.S.: I'll let you in on a little secret: all of the above also applies to closed-source software development within your company. Only now you do owe it to your users to provide this.

Links:

Open Source Security Process Wishlist