After a Friday night party, we found out that someone had accidentally put AWS Access Keys in a public repository (thanks to AWS Security Team for reporting this). Keys didn’t have much access and were of a test environment, but this could have happened with any other key pair with higher privileges. We immediately deactivated them and got the keys rotated everywhere they were being used.

This was not the first time. Things like these have happened in the past with us. Asking friends in the industry reveals that this is a common issue with many engineering teams. Analyzing past incidents like these we learned that whenever this has happened at Grofers, they usually happened because we somehow missed the implications of something we did while trying to build a cool experience for our users. And this can happen to anyone who ends up mishandling “secrets” secret because of a mistake or lack of awareness.

We understood that a big part of the problem is education and started thinking how can we educate our team better but even with all best efforts put in place, things like these can still happen. We realized that education alone is not the solution and we need to put better guards in place — like ones that the AWS Security Team has put for its customers.

How we secured it

We started trying to figure out a way to make AWS keys more secure. In IAM docs (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#policy-vars-infotouse), we came across sourceIP and ec2:SourceInstanceARN conditions.

We have two kinds of users on AWS:

1. Application users — applications using AWS APIs
2. Human users — members of our engineering and product teams using AWS CLI

The only thing we could use was aws:sourceIP. It would be okay for the developers when they are in office, but that would stop them from using keys from their home or anywhere else. We want our developers to have the flexibility of working from anywhere.

We already have a VPN setup to access all of our infrastructure. So we thought why not route all requests to AWS APIs through our VPN itself. This will make the requests AWS APIs with our VPN instance’s IP as the source. Our VPN server has support to push routes to clients when a connection is established. So, we route all requests falling in AWS IP range (https://ip-ranges.amazonaws.com/ip-ranges.json) through our VPN. Now, every request that AWS receives comes through our VPN infrastructure and has the aws:sourceIP of our VPN servers. So, we could now use a policy like this:

"Condition": {
  "IpAddress": {
    "aws:SourceIp": [
      "203.0.113.0/32",
      "23.0.123.0/32"
    ]
  }
}

This restricts usage of AWS Access Key from our VPN only.

We have more than 100 IAM users and more than 300 IAM policies. Some policies are in use by both human users and application users. It’s cumbersome to create new policies and update all the policies to add aws:sourceIP conditions.

According to the documentation here, adding an explicit deny will deny all the requests if it matches that condition.

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {"NotIpAddress": {"aws:SourceIp": [
      "192.0.2.0/24",
      "203.0.113.0/24"
    ]}}
  }
}

Ref: AWS IAM Docs

So we created the above policy and mapped it to IAM group which contained all the human users.

As for application users, we needed a similar thing. Since most of our instances are private, the sourceIP was the internet gateway’s public IP. So we used these IPs for the application users.

We created two IAM Groups, human-users and application-users. And added all the IAM users to these groups according to their type.

Conclusion

Now AWS APIs can be accessed over VPN by human users and from private EC2 instances by application users. This prevents the use of application user’s credentials by human users and vice versa and makes sure that no IAM user’s credentials can be used by anyone outside our private network.

If you are interested in solving problems like this, we are hiring for all roles. Come join us.