6

A Spotify Publisher Was Down Monday Night. The Culprit? A Lapsed Security Certif...

 1 year ago
source link: https://tech.slashdot.org/story/22/06/01/1727218/a-spotify-publisher-was-down-monday-night-the-culprit-a-lapsed-security-certificate
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

A Spotify Publisher Was Down Monday Night. The Culprit? A Lapsed Security Certificate

Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

binspamdupenotthebestofftopicslownewsdaystalestupid freshfunnyinsightfulinterestingmaybe offtopicflamebaittrollredundantoverrated insightfulinterestinginformativefunnyunderrated descriptive typodupeerror

Do you develop on GitHub? You can keep using GitHub but automatically sync your GitHub releases to SourceForge quickly and easily with this tool so your projects have a backup location, and get your project in front of SourceForge's nearly 30 million monthly users. It takes less than a minute. Get new users downloading your project releases today!
×
On Monday night, some Spotify users went to download their favorite podcasts and were met with an error. By Tuesday morning, the issue was resolved. What was the source of the massive disruption impacting some of the platform's biggest producers? An expired security certificate. From a report: The SSL security certificate is what keeps a website secure by enabling encryption, giving it the "s" in HTTPS. For Megaphone, the podcast advertising and publishing platform Spotify acquired in 2020, the certificate expired Monday evening. Shortly thereafter, publishers and listeners for Megaphone-hosted podcasts experienced service disruptions. "Megaphone experienced a platform outage due to an issue related to our SSL certificate," a Spotify spokesperson told NPR. "During the outage, clients were unable to access the Megaphone CMS and podcast listeners were unable to download podcast episodes from Megaphone-hosted publishers. Megaphone service has since been restored." The entire outage lasted for about nine hours, with Megaphone publishing real-time updates of the issue. Some podcast publishers took to Twitter to express their frustration business implications of the outage, according to Verge.
  • This wouldn't have happened if they were using Let's Encrypt.

    • Re:

      Let's Encrypt is not a guarantee and the required 90 day renewal cycle is overly aggressive.

      The last time a cert expired on me, I wrote a little script that gets the expiry date of certs and sends an email reminder 7 days prior to expiry if any of them are about to expire. One of my Let's Encrypt certs almost expired one time after that. My script saw the upcoming expiring cert and notified me. No one else noticed since no outages occurred but there would have been. Let's Encrypt is an improvement but i

      • Re:

        I guess if you have your certificate on a platform that doesn't support auto-renewal, then yeah, it sucks having to update it every 85 days. But you get what you can for free. LE has great automation tools if your platform supports it.
        • Re:

          Auto-renewals can be custom scripted as well. They document their APIs for such use-cases. The aggressive expiry ensures that websites that lose access to their respective domains are not able to pose as that site for long, and private keys that have been compromised are quickly mitigated.
      • Re:

        This problem keeps happening, and it is somewhat baffling. When you get the cert, you know when it expires. If your company operations depend upon this, then add a note to your calendar to renew it. And don't just have a few days reminder, you can have a one month or two of overlap. Especially true if these are self-published certificates that you generate in-house, I'm baffled how those can be allowed to expire only to cause a scramble.

        This isn't really a technological problem, it's an organizational p

        • Re:

          You can even monitor expiry dates with nagios.

        • Re:

          Axway Validation Authority can monitor for expired, fraudulent, or revoked certificates. It can also be configured to monitor for soon-to-expire certificates and send email notifications to the responsible group. The defaults are 14 days before the certificates expire and notifications sent every 60 minutes, but these intervals can be configured as desired.

          Axway Certificate Expiration Warnings [axway.com]

      • Re:

        Just use Certbot. You're not the first person to need LE certificates automatically renewed. Others had the same problem and it's been solved for a while.

      • Re:

        I was certainly being a bit glib in my original post, but my point was that there are really simple solutions to this problem, and yet it keeps happening through a combination of IT staff incompetence and the established CAs making cert registration and renewal way more of a convoluted process than it needs to be.

      • Re:

        The 90 day renewal cycle is "overly aggressive" to make sure your renewal scripts actually work!

    • Re:

      +1! I'm doing Nginx on OpnSense with ACME Client tied into my account on Let's Encrypt and don't even have to worry about renewing certs. Setting up a new server is a breeze, too: set up my DNS, create the reverse proxy in Nginx, create a record in ACME Client, have it get the cert, and bam! All I'm missing is to throw salt on it over my elbow!

      But, in reality, you can blame every mofo fear monger out there who pushed SSL, especially Apple, with their browsers not trusting any certificate that's over 13 m

      • This. Shortening the renewal period wastes everyone's time who can't auto renew due to technical limitations. Rotating certs is the same security fallacy as rotating passwords -- busywork that WILL result in accidental downtime.
    • Re:

      Except that anyone can go to their website (https://www.megaphone.fm/) and look at the certificate, and lo-and-behold it is issued by Lets Encrypt.

    • Re:

      And that's why I keep urging my students to keep a nice calendar to put things on that are due somewhere far in the future.
      Wait, the IT guy probably had a calendar, but then was promoted and all was lost.
    • Re:

      I'm still mad that they turned the internet wireless

  • Major companies do this. Not some silly podcast, but things that cause other things to collapse, etc.

    /. where you hear it first for the millionth time, with more less.
  • Where everything was text based html and no 's'

  • It never ceases to amaze me that some IT people cannot even get basic stuff right. Obviously people that would have struggled to get burger-flipping right, but somehow managed to land an IT job. (With apologies to everybody that can competently flip a burger.)

  • This type of problem can exist only because somebody is making money from it not being solved. If a certificate was valid when it was pinned, there's no reason to not keep accepting it for some period after expiry. (Passports are often accepted for six months post the expiration date as an example.)

    Others have mentioned Lets Encrypt with an API and ninety day renewal which is nice.

    Of course if you pin a certificate and it gets updated, things also break. One would think that there would be a way

    • Re:

      It depends on what you are using the certificate for. At a high level certificates provide a public / private key pair which is used for both encryption and for identity assurance. Proper control of the private key assures that the encryption is solid and that the holder of the private key is who they say they are. If the private key is exposed or compromised, you are no longer assured that encryption will keep the communication private or that the entity on the other end of the network is who they say they

    • Re:

      "(Passports are often accepted for six months post the expiration date as an example.) "

      If you want a certificates to last for 6 months beyond its expiration date... just issue certificates that last 6 months longer in the first place.

      All a 'grace period does' on pinned certificates does is make everything more complicated (code, maintainance, validation) for no real benefit except that the date you set it to expire is not the date it "really" expires now in some cases.

      Right now a certificate is either vali


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK