tl;dr sec is a newsletter about AppSec and scaling security, automated bug finding, conference talk and paper summaries, and useful links from around the web. See past issues here.

Data Driven Bug Bounty

Hey there,

I hope you had a great 4th of July, and that only a few fingers and toes were lost due to fireworks-related celebrations.

This issue includes some key takeaways of

's BSidesSF 2018 talk: "

Data Driven Bug Bounty

." You can

read the full summary here

.

🔗

Links

wrote a nice blog post about

Scaling Appsec at Netflix

. Lots of great perspective, but I wanted to call out 2 things in particular:

• The operational work they do (e.g. bug bounty, pen testing, threat modeling, etc.) is valuable but interrupt-driven, which makes focused engineering efforts hard, so they divide the Appsec team between the two and have a weekly rotation.

• In the past they've focused on automation for vulnerability identification- now they focus on driving/measuring adoption of Security Paved Road practices.

  • This a subtle reframing, but I think a massive and important focus shift.

  • I've been hearing similar thoughts and stories from other forward-thinking Appsec programs, which is why I called it out in my BSidesSF 2019 talk.

Came across a self-reported

spreadsheet on InfoSec salaries

on Twitter. It'd be really interesting to have some charts based on # years experience, location, and income. As expected, SF/Bay Area salaries are towards the high end, but it might be a different story once cost of living has been accounted for 😅

My colleague

pointed me to

GTFOBins

, a curated list of Unix binaries that have functions that can be used to break out of restricted shells, escalate or maintain elevated privileges, transfer files, spawn bind and reverse shells, and facilitate other post-exploitation tasks. Super useful 👍

⭐️

New Summary: Data Driven Bug Bounty

Based on his bug bounty experiences at Twitter and Airbnb, Arkadiy describes:

• How vulnerability metrics can make an AppSec team more effective and which attributes are useful to keep track of

• How to successfully launch and run an effective bug bounty program

While the talk title is about bug bounty, ultimately these ideas and the methodology apply to taking a data-driven approach as an AppSec team in general, regardless of the source of vulnerabilities.

Arkadiy's Context: Twitter and Airbnb

As with anything, it's important to understand someone's context. Arkadiy was involved with both Twitter and Airbnb's bug bounty programs, which had several similarities:

• Both ramped up their program slowly (unpaid soft launch vs unpaid public program + paid private program)

• Both paid for frontline triage (NCC Group vs HackerOne)

• Both had several AppSec engineers on triage rotations

Requirements for Running a Data-driven Bug Bounty Program:

To get started, you need:

1. A breakdown of the different components of the products that vulnerabilities will fall into

2. The teams responsible for each component

3. A bug taxonomy for how you will label vulnerabilities (e.g. XSS, CSRF, broken access controls, etc.)

Other meta info that is useful to track are 4) how long vulnerabilities have been open (SLA) and 5) the source of the vulnerability (e.g. bug bounty, pen test, SAST/DAST tool, etc.).

Show Me the Data - Graphs!

Once you're tracking vulnerability metrics, let's look at some of the useful insights they can give you.

• Know your risk breakdown and focus your energy there- invest in tools and libraries to mitigate the most prevalent and impactful vulnerability classes.

• Inform quarter planning - There’s always more projects you’d like to work on than you have time. This info lets you prioritize projects based on the impact you think it may have and the expected effort required.

• Measure ROI - At the end of the quarter, did the projects you worked on have the outcomes you wanted? If not, why? Maybe it was the wrong approach or maybe you need to invest additional effort.

• Track vulnerability ticket SLAs - Which teams are consistently missing SLAs? Which teams are trending better or worse over time?

• Hold teams accountable - When you see a team is working more slowly, investigate to determine why. Teams have their own priorities and roadmap. When you can bring objective data like this to discussions it can provide valuable context and motivation for the team to improve.

When you look at the data like this, clear trends become apparent and next actions become obvious. In this case, the Airbnb AppSec team started partnering closely with the team that had the largest number of open vulnerabilities to help them improve.

This information helps drive conversations forward, as when you reach out to development teams you have data backing up your requests, not just qualitative opinions.

Measure improvement (or lack of improvement) over time - This is really important for the business decisions you’re making. If things aren’t working, do you need to invest more? Also, share the wins!

Use data to drive business goals - Data can help you make the case for more resources, changing current resource allocation, or recommending other big picture organizational changes.

Raising the Security Team's Visibility and Getting Organizational Buy-in

This figure is great because it requires no security domain knowledge to interpret. This means it can be shared widely, in presentations, a security newsletter, a dashboard, or an internal wiki page.

Raising the security team's visibility shows your non-security colleagues the work you're doing and the value you're adding. By being more visible, you build social capital and gain executive and leadership support.This is especially useful, for example, if in the future you need to make a non-transparent change in a process or development workflow that adds some friction - you'll get less pushback because the security team and its wins have been visible.

You can watch for changes in this data over time for purposes like:

• Tuning scanners - If a greater percentage of scanner reports are getting marked as invalid, perhaps you need to tune, replace, or get rid of this type of tool.

• Bug bounty program health - Has there been a decrease in the number of bug bounty submissions? Investigate if the rewards haven’t been appropriate, maybe you haven’t been responding quickly enough, etc.

• Correlate vulnerability source and volume with impact. For example, maybe scanning tools are reporting many issues, but they all tend to be low severity, whereas pen tests aren’t reporting many issues but the ones they do are high impact.

How to launch a bug bounty program:

1. Start with a pen test and assess yourself so you’re not flooded with submissions.

2. Launch as a private program with few researchers and limited scope.

3. Increase the number of researchers and program scope slowly, tuning your workflow along the way.

4. Go public when you’re ready - when your process is streamlined, there are 100s of researchers in your private program, and you’ve seen a down tick in submissions.

The

core health metrics

of an effective bug bounty program are: the time it takes to first respond to a researcher, time to triage, time to bounty, and time to resolution.

Time to response

and

time to bounty

are overall the most important.

Tips for Interacting with Dev Teams

Give positive reinforcement

- Don’t always be the bearer of bad news. Celebrate good behavior so teams want to work with you. For example, the Airbnb AppSec team threw a cupcake party for a specific team that hit a sustained 0 open vulnerabilities past SLA.

Reach out to teams with tact

; otherwise, people can get defensive. Instead, be positive and collaborative.

In summary

Vulnerability metrics allow you to:

• Prioritize investing your time and money where it'll make the biggest impact, for example, by targeting the most common and impactful bug classes, as well as choosing to invest more or less in efforts such as security partnerships, pen tests, tools, bug bounty, etc.

• Determine whether the projects you've pursued have actually helped, and if so, how much.

• Focus your time working with teams who need the most help.

• Increase the security team's visibility in your company and build valuable social capital with executive leadership.

✉️

Wrapping Up

Have questions, comments, or feedback? Just reply directly, I'd love to hear from you.

Also, if there's anyone who you think would find this newsletter interesting or useful, I'd really appreciate if you'd forward it to them.

Thanks for reading!

Cheers,

Clint

|