Design by Asif Jamal

When it comes to security, we always take it as the utmost priority. We strongly believe that “Security shouldn’t be treated as an after-thought”, it should be brought as close to engineers and as early in SDLC.

Aside from the general guidelines put forth in the CIS benchmark for all around information security, we have automated infrastructure scans for audit, compliance, automated penetration tests including both DAST and SAST, performing manual pen-testing as well and having strong firewalls at multiple layers.

We are immensely proud of the infrastructure security that we are able to build but there is a much bigger challenge that companies face — “Classic” security systems are more tend to be reactive in nature.

We realized that most of the application security, we have in place is more of a detective measure. So we started working on a more powerful preventative strategy and approach to application security.

We have been implementing various ideas and solutions to improve the existing security state of SDLC and bring security as close to developers as possible. We wanted to implement security adherence and adoption amongst our engineers.

In order to do this well, the most logical place security can be checked are code reviews. Since at the time when we were implementing this, we didn’t have a central pipeline so we thought of building our tool which can listen to any new PR raised.

That is when we came across — GitHub Apps. They come with built-in webhooks and narrow, specific permissions. When you set up your GitHub App, you can select the repositories you want it to access.

For example, you can set up an app called MyGitHub that writes issues in the octocat repository and only the octocat repository. So we created a Github App — G-Shield, which can be integrated with repositories, have read access to PRs and comment permissions, and configure it to post events to a pre-configured webhook URL.

When any event is triggered, it sends an HTTP POST request to the webhook. This webhook can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server.

Thiswebhook’s configured URL is the heart of our “Continuous Github PR Hacking”. This is from where our modular, highly customized, automated security checks are performed.

So the G-Shield continuouslysends the input received from Github events i.e the PR events to the backend app which performs all the checks, which we want it to deliver and take specific actions based on particular conditions.

This is how the basic architecture of G-Shield looks like. We are using Celery as an asynchronous task queue system for queueing the Github PR events to process.

To perform source code scanning, we have setup Sonarqube (configured hardening and meticulous tuning) and git secrets for detecting PR containing hardcoded credentials and preventing devs from committing passwords and other sensitive information to a git repository. Further integrating it with Jenkins via declarative pipeline.

And then we have independent modules which perform other security checks like -

1. Runs Hadolint for a smarter Dockerfile linter that helps build best practice Docker images.
2. Detect PR having dockerfile to check for sensitive volume mount — We don’t allow images to be built whose Dockerfile specifies a sensitive host path as a volume mount — meaning, we scan the Dockerfile for volume mounts like /proc or /,and others.
3. Check whitelist of base images —Function detects if PR has Dockerfiles and check FROM directives to find whether the base image is a base Docker standard library image or pointing to a smart whitelist of base images. We call it smart because it contains some inhouse images which keep on expanding as we create more.

The Github app pipeline is highly customized which can help us in adding more security checks easily as we require. Any of the results which are failing security checks is commented on the PR as a warning.

Note: We can also block the PR from getting merged till the security issue detected by G-Shield are not resolved.

It has brought in the following improvements:

1. Early Checks — Now security checks are performed as soon as any PR is raised and the result is posted on PR as a comment to review.
2. Highly Flexible —The security checks are very modular. We can add more checks as we want and configure G-Shield to perform response based action.
3. Completely Automated — Automation is the key/let the machines do the work.

What next?

As we scale up our platform, bringing more and more technologies into the system, the chances of having a security loophole become more and more real and hence the challenges to keep our infrastructure secure.

Since security is a continuous process, we continue working towards strengthening the security of our infrastructure and apps through tools and processes. Keep following us to know more about them. Stay tuned!

Stay tuned! 👋

More on Grofers Security → How We Sustain DNS Outages at Grofers

Credentials leaked in public? Here’s what Grofers implemented to prevent such mishaps! ←

Avinash Jain is the Lead Infrastructure Security Engineer at Grofers and he loves to break application logic and find vulnerabilities in them.

Assem Shrey is a Security Engineer at Grofers and he loves to tinker with applications (mobile & web), automating stuff, and participates in CTFs.