A collection of Kubernetes failure stories

Kubernetes Failure Stories

A compiled list of links to public failure stories related to Kubernetes. Most recent publications on top.

• 10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You - Datadog - KubeCon Barcelona 2019
  • involved: CoreDNS, ndots:5 , IPVS conntrack, imagePullPolicy: Always , DaemonSet, NAT instances, latest tag, API server OOMKill , kube2iam, cluster-autoscaler, PodPriority, audit logs, spec.replicas , AWS ASG rebalance, CronJob, Pod toleration, zombies, readinessProbe.exec , cgroup freeze, kubectl
  • impact: unknown, API server outage, pending pods, slow deployments
• How Spotify Accidentally Deleted All its Kube Clusters with No User Impact - Spotify - KubeCon Barcelona 2019
  • involved: GKE, cluster deletion, browser tabs, Terraform, global state file, git PRs, GCP permissions
  • impact: no impact on end users
• Kubernetes Failure Stories - Zalando - KubeCon Barcelona 2019
  • involved: Skipper-Ingress, AWS, OOMKill , high latency, CronJob, CoreDNS, ndots:5 , etcd, CPU throttling
  • impact: multiple production outages
• Oh Sh*t! The Config Changed! - Pusher - KubeCon Barcelona 2019
  • involved: AWS, nginx, ConfigMap change
  • impact: production outage
• Misunderstanding the behaviour of one templating line - Skyscanner - blog post 2019
  • involved: HAProxy-Ingress, Service VIPs, Golang templating
  • impact: Significantly increased latency, 5XXs thrown from some services
• All pods scheduled to same failing host - Moonlight - postmortem 2019
  • involved: Google Kubernetes Engine, scheduler, anti-affinity rules
  • impact: major production outage, 100% traffic loss
• The shipwreck of GKE Cluster Upgrade - Loveholidays GKE - blog post 2019
  • involved: GCP Ingress, GKE Cluster, nodes
  • impact: critical drop in pod availability, loss of ingress, 2 hour maintenance lasting 7 hours
• Breaking Kubernetes: How We Broke and Fixed our K8s Cluster - Civis Analytics - blog post 2019
  • involved: AWS, kops, large clusters, batch jobs infrastructure, Datadog, API server, DNS, CPU throttling
  • impact: production outage
• Let's talk about Failures with Kubernetes - Zalando - Hamburg meetup 2019
  • involved: AWS, NotReady nodes, ELB dynamic IPs, Ingress, API server, CronJob, CoreDNS, OOMKill , kubelet memory leak, CPU throttling
  • impact: production outages
• Total DNS outage in Kubernetes cluster - Zalando - postmortem 2019
  • involved: AWS, DNS, CoreDNS, OOMKill , ndots:5 , HTTP retries
  • impact: production outage
• Maximize learnings from a Kubernetes cluster failure - NU.nl - blog post 2019
  • involved: AWS, NotReady nodes, SystemOOM , Helm, ElastAlert, no resource limits set
  • impact: user experience affected for internally used tools and dashboards
• Kubernetes Load Balancer Configuration - Beware when draining nodes - DevOps Hof - blog post 2019

  externalTrafficPolicy
  

• On Infrastructure at Scale: A Cascading Failure of Distributed Systems - Target - Medium post January 2019
  • involved: on-premise, Kafka, large cluster, Consul, Docker daemon, high CPU usage
  • impact: development environment outage
• How NOT to do Kubernetes - Sr.SRE Medya Ghazizadeh - Google - Cloud Native Meetup Sep 2018
  • involved: public container registery, ingress wild card, image size, replica count, 12factor
  • impact: security, stablity of clusters.
• Running Kubernetes in Production: A Million Ways to Crash Your Cluster - Zalando - DevOpsCon Munich 2018
  • involved: AWS, Ingress, CronJob, etcd, flannel, Docker, CPU throttling
  • impact: production outages
• Outages? Downtime? - Veracode - blog post 2018
  • involved: AWS, AWS IAM, region migration, kubespray, Terraform, pod CIDR
  • impact: QA/dev cluster outage
• NRE Labs Outage Post-Mortem - NRE Labs - blog post 2018

  livenessProbe
  

• A Perfect DNS Storm - Toyota Connected - blog post 2018

  ndots:5
  

• Kubernetes and the Menace ELB, the tale of an outage - Turnitin - blog post 2018

  NotReady
  

• Moving the Entire Stack to K8s Within a Year – Lessons Learned - ThredUP - DevOpsStage 2018

  livenessProbe
  

• AirMap Platform Service Outage - AirMap - incident report 2018

  NotReady
  

• Anatomy of a Production Kubernetes Outage - Monzo - KubeCon Europe 2018

  NullPointerException
  

• Stories from the Playbook - Google - KubeCon Europe 2018
  • involved: GKE, etcd, Docker daemon, Image registry, dnsmasq vulnerabilities
  • impact: production outage
• 101 Ways to "Break and Recover" Kubernetes Cluster - Oath/Yahoo - KubeCon Europe 2018

  NotReady
  

• 101 Ways to Crash Your Cluster - Nordstrom - KubeCon North America 2017

  NotReady
  

• Major Outage: Current account payments may fail - Monzo - Monzo Community post 2017

  NullPointerException
  

• Fallacies of Distributed Computing with Kubernetes on AWS - Zalando - AWS User Group Hamburg October 2017
  • involved: AWS, unhealthy nodes, Ingress, CronJob
  • impact: production outage
• Search and Reporting Outage - Universe - incident report 2017

  RestartPolicy
  

• Our First Kubernetes Outage - Saltside - blog post 2017

  NotReady
  

• Our Failure Migrating to Kubernetes - Saltside - blog post 2017
  • involved: AWS, kops, ELB, BackendConnectionErrors , LoadBalancer service
  • impact: aborted application migration
• SaleMove US System Issue - SaleMove - incident report 2017

  A
  

Why

Kubernetes is a fairly complex system with many moving parts. Its ecosystem is constantly evolving and adding even more layers (service mesh, ...) to the mix. Considering this environment, we don't hear enough real-world horror stories to learn from each other! This compilation of failure stories should make it easier for people dealing with Kubernetes operations (SRE, Ops, platform/infrastructure teams) to learn from others and reduce the unknown unknowns of running Kubernetes in production. For more information, see the blog post .

Contributing

Please help the community and share a link to your failure story by opening a Pull Request! Failure stories can be anything like blog posts, conference/meetup talks, incident postmortems, tweetstorms, ...

I would also be glad to hear about your failure stories on Twitter: my handle is @try_except_