Snyk @ Snyk: Enabling Kubernetes RBAC for Snyk’s Developers

As Uncle Ben once said, “With great power comes great responsibility.” This is also true of the Kubernetes API. It is very powerful, and you can build amazing things on top of it, but it comes with a price—a malicious user can also use the API to do bad things.

Enter Kubernetes RBAC (role based access control), which enables you to use the API in a controlled manner by granting only required privileges needed, following least privilege principle. 

But this just moves the problem somewhere else: now we need to have good processes around RBAC resources or a simple RBAC mistake could have a serious security impact. Take the example of a developer ‘accidentally’ granting the cluster admin role to the default service account in the default namespace. Oh, the horror!

Photo by https://unsplash.com/@arnykoor

The paved road approach to Kubernetes security

Ideally, we would like to allow every developer to create RBAC permissions as they desire, while still having guardrails in place for protection and to block things that are insecure. One way to achieve that is by enforcing a code review by AppSec for any change that includes RBAC resources. While this might work, it has some downsides:

• AppSec becomes a blocker, slowing down developers.
• Security becomes “magic” that only AppSec knows, as there are no published guidelines to what AppSec is looking for when reviewing such changes. 
• We end up deepening silos, and preventing developers from owning security responsibilities, which further compounds the problem of AppSec being a bottleneck and slowing down developers.

The Kubernetes RBAC security checklist

At Snyk, we faced similar issues. We wanted to empower our developers and let them use the Kubernetes API without slowing them down. We started by compiling a list of security requirements for RBAC rules. The rules list is pretty short and based mainly on the CIS Kubernetes benchmarks:

• No wildcard resources or verbs
• No usage of built-in rules, as they grant too many permissions
• No bindings to service accounts in default namespace (which should be empty) or kube-system namespace (which runs sensitive workloads)
• No binding to default service account
• No usage of dangerous permissions, like create pods or read secrets

With our list compiled, we can do a few things with it:

• Publish it as our internal guidelines and require that all RBAC objects adhere to this list. Now the criteria are not “magic”.
• Ask our developers and security engineers to document any violation of this list as a security risk, and inform AppSec of such violations.
• Solicit feedback and invite all the developers to contribute and update the list so we break down the silos.
• AUTOMATION. After all, once we have a list, we want to make it ridiculously easy to test. 🙂

Automating Kubernetes configuration checks with Snyk IaC

In late 2020, Snyk announced a new product: Snyk Infrastructure as Code (Snyk IaC). Snyk IaC has a built-in set of rules we can use to test our IaC code, including Kubernetes manifests (Terraform, too, but that’s for another blog). One of the perks of working for Snyk is early access to any feature, and, even better, the ability to contribute back to the product. (By the way, if that kind of perk excites you, we are hiring, and looking for AppSec folks!)

So, after compiling our list, we reached out to the Snyk IaC team, discussing our Kubernetes RBAC checklist with them, and voila! We have five new checks in Snyk IaC to help us automate our security checks and make sure AppSec isn’t a bottleneck!

Making IaC security ridiculously easy for Snyk’s developers

Snyk IaC has the ability to scan via GitHub Actions and output results in Sarif format, which could be integrated with GitHub Security. But we wanted to take it a step further and show violations directly in our PRs, which Snyk IaC doesn’t do (yet). Our AppSec group did this by writing our own GitHub integration so that any change to our Kubernetes manifest files are automatically scanned with Snyk when committed, and if there is a violation, the developer will get a nice warning on the PR:

Now, all our developers are free to make RBAC changes without pausing to get AppSec involved and in AppSec can rest easier, knowing that Snyk is watching our back.

Want to have something similar for your team? Checkout the example repo here, and make sure you have a Snyk account—you can get started with IaC for free!