Authors: James Cha, Vickie Li, Shashank Mirji, and Frank Filho

Let’s face it, there is no such thing as 100% bug-free code. And there is particularly no such thing as 100% security. That’s why a security program that catches security vulnerabilities is an important part of a well-rounded engineering organization.

Instacart’s Security Team works to mature scalable tooling to build guardrails for the Instacart engineering team. However, security blindspots inevitably exist — or at least, savvy security teams have to assume so. Our bug bounty program acts as the canary mechanism to let us know what we don’t know and what escaped our preventative/detective measures.

What does it take to run a standout Bug Bounty program?

Running a successful bug bounty program involves many moving parts, and organizations often struggle to maintain the balance between the stakeholders of a bug bounty program — researchers, bug bounty platform, and security engineers.

Over the past few years, Instacart’s bug bounty program has grown exponentially in terms of reports and payout amounts. As Instacart grows, we needed a way to grow our bug bounty program to keep pace with our engineering infrastructure. This article is about how Instacart matured our bug bounty program over the years. And our surveys have shown that researchers really enjoy hunting on our program! Based on our recent survey results, about 80% of our researchers would recommend the Instacart’s Bug bounty program to other researchers/friends.

We built a successful bug bounty program by:

• Introducing broad and well-defined scopes. So that the researchers can make the best use of their time and let their creativity out for chaining the found vulnerability across multiple Instacart domains.
• Helping researchers succeed (and save time!)by providing recon data like the API documents to the researchers and giving priority/impact rating on the in-scope domains.
• We payout competitively by benchmarking with the highest paying programs
• Rewarding fairly and transparently. We explain to the researcher why we decided on a payout amount
• Rewarding and retaining top performers by treating them well. We issue bounties fairly and honor their concerns

Building a Boutique VIP Invite-only Program

We run two bug bounty programs:

• A public program open to everyone
• VIP program for invited researchers only

Both programs are hosted on HackerOne. The scope for both programs is the same and includes most of Instacart’s core assets, including our web platforms, mobile apps, APIs, and any system or service within *.instacart.com or *.instacart.tools. Our public program is one of the top paying programs on HackerOne and is open to all researchers.

However, we invite the best performing researchers on Instacart’s public program to join the VIP program, through which they are provided with access to internal resources like API documentation and DNS zone file records to speed up their technical research.

These researchers have typically reported a high-impact finding before and have been working on our program over the years. These researchers help look deep into our technology stack and understand the business logic very well.

To acknowledge the extended effort and expertise in our platform, the maximum reward amount for the high and critical severity finding are increased significantly compared to the public program.

Bug Bounty Program Progress

Over the past two years, we have received 1400+ submissions and rewarded over $500,000 in bounties since 2020. We launched a VIP program in 2021 to attract top-tier researchers, resulting in the program becoming the 2nd highest ranked VIP program on HackerOne, with total bounty payouts above $200,000. We saw a 300% increase in the critical & high severity security bugs reported in the last two years thanks to increased participation in the program.

Here’s a summary of the number of submissions we received and the payout amount since 2018:

2021 was our biggest year for bounties; we experienced a 300% increase in the critical & high severity security bugs reported. This was due to the launch of our VIP program and the overall increase in the bounty amounts. The number of active researchers within our programs also increased by over 30% in 2021. We continue to see a similar trend in 2022, and based on our forecast, we anticipate rewarding around $250K+ bounties by the end of the year.

How we reward bounties

At Instacart, we aim to value our researchers by rewarding bounties fairly. When it comes to determining how much the bounty payout amount should be, the security team first looks at the overall severity of the bug found to determine the minimum and maximum payout amount.

The overall severity of the risk is calculated by examining the likelihood and impact of the vulnerability reported. Simply put, the easier the bug is to exploit, and the more severe its potential impact, the higher the reward.

Bug classification

We usually use the vulnerability’s bug class to guide bounty decisions. Each vulnerability type has a typical severity level; for example

Impact

These are all guidelines we use to make an objective decision, and sometimes submissions are more nuanced and require adjustments to our standard policy. As a rule of thumb, for non-standard submissions, the bigger the impact, the bigger the payout. For example, to determine payouts, we may use criteria like whether the finding results in a code or configurational change or the scale of the theoretical amount of data that might be exposed through the vulnerability.

Historical payouts

To calibrate further, we also look at historical payout amounts for a vulnerability of the same category and impact statement. To be fair to all researchers working on our bounty program, we sometimes retroactively pay out additional boun ties when we find out about the bug’s impact later on or if we have re-calibrated the payout amount of a similar bug.

For example, we used to pay $1,000 for an unauthenticated Cross-Site-Scripting (XSS) bug and $3,000 for an authenticated XSS bug. But recently, we decided that those two bug types have a very similar, if not the same, impact. So we retroactively went back to the former bug reportings and matched them as $3,000.

CVSS score

However, sometimes it’s unclear how severe the finding could be. In that case, we will use the Common Vulnerability Scoring System (CVSS) Calculator to calculate a bug’s severity. This calculator is used to measure the severity of a vulnerability by carefully classifying its likelihood and impact.

During our weekly bug bounty meeting, we discuss all of the above as a group, try to sum up all the potential impact the finding can have, and agree upon the payout amount. This is when the security team members conduct a technical impact assessment of the bug found and try to explore further escalation or exploitation paths that the researcher may not have been aware of. In many cases, researchers may receive a much higher amount than expected.

Bug bounty guide

The security team at Instacart strives to create a rewarding, fair, and exciting bug bounty program for our researchers.

In our efforts to make our program more researcher friendly, we aim to make hunting and reconing easier for researchers in the future. In September 2019, we launched a bug bounty guide to our private VIP program. We’re now making it available to anyone at https://bug-bounty-guide.instacart.com. Instacart’s Bug Bounty Guide is a collection of hand-written resources with a high-level overview of how our systems work and automatically generated API documentation of our endpoints, expected inputs, and outputs.

Researcher Spotlight

Justin Gardner (@Rhynorater), based in Japan, leads the leaderboard and was awarded our highest bounty.

“I’ve really enjoyed working with the Instacart team — Not only do they offer great bounties, but they also triage quickly and issue bounties not long after. There are also great things waiting for you if you happen to make it into the VIP program!”

Contributors to the Bounty Program — Shoutouts!

Ankit A, David G, Frank F, James C, Kyriakos G, Shashank M, Tim L, Vickie L.