Open sourcing the Kubernetes security audit

Last year, the Cloud Native Computing Foundation (CNCF) began the process of performing and open sourcing third-party security audits for its projects in order to improve the overall security of our ecosystem. The idea was to start with a handful of projects and gather feedback from the CNCF community as to whether or not this pilot program was useful. The first projects to undergo this process were CoreDNS , Envoy and Prometheus . These first public audits identified security issues from general weaknesses to critical vulnerabilities. With these results, project maintainers for CoreDNS, Envoy and Prometheus have been able to address the identified vulnerabilities and add documentation to help users.

The main takeaway from these initial audits is that a public security audit is a great way to test the quality of an open source project along with its vulnerability management process and more importantly, how resilient the open source project’s security practices are. With CNCF graduated projects especially, which are used widely in production by some of the largest companies in the world, it is imperative that they adhere to the highest levels of security best practices.

Findings

• Kubernetes Security Review
• Attacking and Defending Kubernetes Installations
• Whitepaper
• Threat Model

Since the pilot has proven successful, CNCF is excited to start offering this to other projects that are interested, with preference given to graduated projects.

With funds provided by the CNCF community to conduct the Kubernetes security audit, the Security Audit Working Group was formed to lead the process of finding a reputable third party vendor. The group created an open request for proposals, taking responsibility for evaluating the submitted proposals and recommending the vendor best suited to complete a security assessment against Kubernetes, bearing in mind the high complexity and wide scope of the project. The working group selected two firms to perform this work: Trail of Bits and Atredis Partners . The team felt that the combination of these two firms, both composed of very senior and well-known staff in the information security industry, would provide the best possible results. 

The Security Audit Working Group managed the audit over a four month time span. Throughout the course of this work, a component-focused threat model of the Kubernetes system was conducted Working with members of the Security Audit Working Group, as well as a number of Kubernetes SIGs,this threat model reviewed Kubernetes’ components across six control families:

• Networking
• Cryptography
• Authentication
• Authorization
• Secrets Management
• Multi-tenancy

Since Kubernetes itself is a large system, with functionality spanning from API gateways to container orchestration to networking and beyond, the Third Party Security Audit Working Group, in concert with Trail of Bits and Atredis Partners, selected eight components within the larger Kubernetes ecosystem for evaluation in the threat model:

• Kube-apiserver
• Etcd
• Kube-scheduler
• Kube-controller-manager
• Cloud-controller-manager
• Kubelet
• Kube-proxy
• Container Runtime

The assessment yielded a significant amount of knowledge pertaining to the operation and internals of a Kubernetes cluster. Findings and supporting documentation from the assessment has been made available today, and can be found here .

There were a number of Kubernetes-wide findings, including:

1. Policies may not be applied, leading to a false sense of security.
2. Insecure TLS is in use by default.
3. Credentials are exposed in environment variables and command-line arguments. 
4. Names of secrets are leaked in logs.
5. No certificate revocation.
6. seccomp is not enabled by default.

Guidance was provided to promote further assessments and discussion of Kubernetes from the perspectives of cluster administrators and developers:

Recommendations for cluster administrators included:

• Attribute Based Access Controls vs Role Based Access Controls
• RBAC best practices
• Node-host configurations and permissions
• Default settings and backwards compatibility
• Networking
• Environment considerations
• Logging and alerting

Recommendations for Kubernetes developers included:

• Avoid hardcoding paths to dependencies
• File permissions checking
• Monitoring processes on Linux
• Moving processes to a cgroup
• Future cgroup considerations for Kubernetes
• Future process handling considerations for Kubernetes

This audit process was partially inspired by, the Core Infrastructure Initiative (CII) Best Practices Badge program that all CNCF projects are required to go through. This badge, provided by the Linux Foundation, is a way for open source projects to show that they follow security best practices. Consumers of the badge can quickly assess which open source projects are following best practices and as a result are more likely to produce higher-quality secure software.

Finally, we hope that open sourcing our security audits and process, we inspire other projects to pursue them in their respective open source communities.

The CNCF wishes to thank the members of the Security Audit Working Group, as well as Kubernetes community who assisted in the threat model and audit work:  

Aaron Small, Google          Security Audit Working Group member

Craig Ingram, Salesforce    Security Audit Working Group member

Jay Beale, InGuardians      Security Audit Working Group member

Joel Smith, Red Hat           Security Audit Working Group member