Last Week's Internet Calamity That Wasn't

This is a story about something that could have gone wrong on the internet this week but instead turned out mostly OK. How often can you say that?

Around nine o’clock on the East Coast on Friday, February 28, bad news arrived on the doorstep of Let’s Encrypt. An arm of the nonprofit Internet Security Research Group,Let’s Encrypt is a so-called certificate authority that lets websites implement encrypted connections at no cost. A CA parcels out digital certificates that essentially vouch that a website isn't an imposter. That cryptographic guarantee is thebackbone of HTTPS, the encrypted connections that keep anyone from intercepting or spying on your interactions with websites.

Those certificates expire after a set amount of time; Let's Encrypt caps its certificates at 90 days, at which point a site operator has to renew. It's a largely automated process, but if a site doesn't have an active certificate your browser will notice, and either cripple or outright break it. Think of it sort of like updating the registration on your car every year. If your tags expire, you'll get pulled over.

Let's Encrypt's work is technical, and happens in the background. But in a few short years it has helped make the internet much more secure on a fundamental level. Plenty of companies offer security certificates; Let’s Encrypt just took the audacious step of making them free. A week ago, it issued its billionth certificate.

But that ubiquity also means that when a pebble drops in the middle of Let’s Encrypt’s pond, the ripples can travel a long way. On February 28, the pebble was a bug that threatened to effectively render three million sites nonfunctional in a matter of days.

The flaw itself? Relatively minor in the grand scheme of the internet. Let's Encrypt uses software called Boulder to make sure that it's allowed to issue a certificate to a site. (Some high-value targets, like banks, specify that they'll only accept certificates from a particular CA. Let's Encrypt has solid security, but some paid certificate authorities offer warranties in the event anything goes wrong, as well as other upgrades. It's the difference between, say, having a strong deadbolt and adding renter's insurance.) Boulder confirms that Let's Encrypt is honoring those preferences when it first issues a certificate and again 30 days later. Or at least, it’s supposed to; the bug meant it was skipping the second check. And that’s a big no-no.

The actual security implications of that backend hiccup were minimal, says ISRG executive director Josh Aas. At the same time, Let’s Encrypt couldn’t let a bug that affected 2.6 percent of its active certificates—3,048,289 in all, when it confirmed the issue—linger indefinitely. “The severity of the bug here is not very high,” says Aas. “But these three million certificates were issued in a noncompliant way. We have an obligation to revoke them.”

That obligation stems from the Certification Authority Browser Forum, or CA/B, an industry group that sets strict standards about the use of certificates. In this case, those standards gave Let's Encrypt a five-day window to come back into compliance, which would entail revoking every certificate that was affected by the bug. The alternative for Let's Encrypt was ignoring the CA/B and letting it slide, but that was really no option at all.

“They did the right thing. The CA/B sets these rules and has fairly strict requirements, which you want. When a person or computer talks to another computer, you want to make sure they’ve met some identity criterion,” says Kenneth White, security principle at MongoDB, a massive database provider that uses Let’s Encrypt. “You can’t be mostly correct. You’ve got to follow the guidelines for how to enforce these things.”