A​ Pragmatic Approach for Internal Security Partnerships

A​ Pragmatic Approach for Internal Security Partnerships

Scott Behrens, Senior AppSec Engineer, Netflix 
Esha Kanekar, Senior Security Technical Program Manager, Netflix
abstract slides video

How the Netflix AppSec team scales their security efforts via secure defaults, tooling, automation, and long term relationships with engineering teams.

The Early Days of Security at Netflix

In the beginning, the Netflix AppSec team would pick a random application, find a bunch of bugs in it, write up a report, then kick it over the wall to the relevant engineering manager and ask them to fix it. Their reports had no strategic recommendation section, just a list of boilerplate recommendations/impact/description of vulns. Essentially, they were operating like an internal security consulting shop.

This approach was not effective - vulns often wouldn’t get fixed and the way they operated caused relationships with dev teams to be adversarial and transactional. They were failing to build long term relationships with product and application teams. They were focusing on fixing individual bugs rather than strategic security improvements.

Further, dev teams would receive “high priority” requests from different security teams within Netflix, which is frustrating, as it was unclear to them how to relatively prioritize the security asks and the amount of work was intractable.

Enabling Security via Internal Strategic Partnership

Now, the AppSec team aims to build strong, long term trust-based relationships with dev teams.

They work closely with app and product teams to assess their security posture and identify and document investment areas: strategic initiatives that may take a few quarters, not just give dev teams a list of vulns.

Security Paved Road

The Netflix AppSec team invests heavily in building a security paved road: a set of libraries, tools, and self-service applications that enable developers to be both more productive and more secure.

For example, the standard authentication library is not only hardened, but it also is easy to debug and use, has great logging and gives devs (and security) insight into who’s using their app, and in general enables better support of customers who are having issues, whether it’s security related or not.

Tooling and Automation

The Netflix AppSec team uses tooling and automation in vuln identification and management as well as application inventory and risk classification.

This information give them valuable data and context, for example, to inform the conversation when they’re meeting with a dev team, as they’ll understand the app’s risk context, purpose, etc. and be able to make better recommendations.

Intuition and organizational context are still used though. They’re data informed, not solely data driven.

Netflix’s 5-Step Approach to Partnerships

1. Engagement identification

• Identify areas of investment based on factors like enterprise risk, business criticality, sensitivity of data being handled, bug bounty submission volume, overall engineering impact on the Netflix ecosystem, etc.

• Ensure that the application team is willing to partner.

2. Discovery meeting(s)

• Basically a kick off and deep dive meeting with the app team.

• What are some of their securiity concerns? What keeps them up at time?

• Set context with stakeholders: we’re identifying a shared goal, not forcing them to a pre-chosen security bar.

• Make sure security team’s understanding of the app built via automation aligns with what the dev team thinks.

3. Security review

• Based on info collected via automation and meeting with the dev team, the security team now knows what security services should be performed on which app and part of their landscape.

  • You don’t need to pen test or do a deep threat model of every app. This process can be more holistic, sometimes over a set of apps rather than a specific one.

• Work with other security teams to collect their security asks for the dev team. This full list of security asks are then priorized by the security team and consolidated into a single security initiatives document.

4. Alignment on the Security Initiatives Doc

• Discuss the security initiatives document with the dev team to ensure there is alignment on the asks and their associated priorities.

5. On-going relationship management and sync-ups

• After aligning on the security asks, the dev teams make the initiatives part of theiri roadmap.

• The sync ups are the key to maintain the long term relationship with partnering teams. Meetings may be bi-weekly or monthly and the security point of contact may join their all-hands, quarterly meeting plans, etc.

• These meetings are not just to track that the security initiatives are on the app team’s roadmap, but also to ask if they have any blockers, questions, or concerns the security team can help with.

• What’s going on in their world? What new things are coming up?

Automation Overview

Before going through a case study of this partnership process, let’s quickly cover some of the tooling and automation that enables Netflix’s AppSec team to scale their efforts.

Application Risk Rating: Penguin Shortbread

Penguin Shortbread performs an automated risk calculation for various entities including:

• Finding all apps that have running instances and are Internet accessible, by org.

• Can see which apps are using security controls, like SSO, app-to-app mTLS and secrets storage, and if they’re filled the security questionnaire (Zoltar).

The allows the security team to walk into a meeting and celebrate the team’s wins, “Hey, it looks like all of your high risk apps are using SSO, that’s great. You could buy down your risk even further by implementing secret storage in these other apps.”

Application Risk Calculator

This gives a ballpark understandinig of which apps the security team should probably look at first, based on properties like: is it Internet facing? Does it have an old OS or AMI? Which parts of the security paved road is it using? How many running instances are there? Is it running in a compliance-related AWS account like PCI?

Vulnerability Scanning: Scumblr

Scumblr was originally discussed by Netflix at AppSec USA 2016 (video) and was open sourced. It has since been changed heavily for internal use, and in general is used to run small, lightweight security checks against code bases or simple queries against running instances.

Security Guidance Questionnaire: Zoltar

Zoltar keeps track of the intended purpose for different apps and other aspects that are hard to capture automatically. As devs fill out the questionnaire, they’re given more tailored advice for the language and frameworks they’re using, enabling the dev team to focus on things that measurably buy down risk.

Case Study: Kicking Off a Partnership with the Content Engineering Team

Discovery Meeting Preparation: Understand Their World

The team they’re meeting with may have 100-150 apps. The security team uses automation to come to the meeting as informed as possible:

• Risk scoring - Based on the data the apps handle, if they’re Internet facing, business criticality, etc.

• Vulnerability history - From internal tools, bug bounty history, pen tests, etc.

• Security guidance questionnaire - Questionnaire that the dev team fills out.

Discovery Meeting

The main point of meeting is to build trust and show that the security team has done their homework. Instead of coming in and asking for a bunch of info, the security team is able to leverage their tooling and automation to come in knowing a lot about the dev team’s context - their apps, the risk factors they face, and the likely high risk apps they should focus on. This builds a lot of rapport.

If app teams aren’t receptive, that’s OK, the security team will circle back later.

Security Review

Perform holistic threat modeling/security services
This is different than a normal securiity review, as 100-200 apps may be in scope. They threat model a team’s ecosystem, not just a single app. Because the security team is informed of the teamm’s apps annd risks, they can narrow the security help they provide and what they recommend in a customized way.

By investing in secure defaults and self-service tooling, the security team can focus on things that add more value by digging into the harder problems, rather than walking in and saying, “You need to use X library, you need to scan with Y tool,” etc.

Collect and prioritize context/asks from other Security Teams
So that the dev team doesn’t need to worry about the 6 - 12 security subteams. The various security teams align on which security asks they want the dev team to prioritize.

Document the security asks into a security initiatives doc
Put all of the asks and related useful meta info into a standalone doc for easier tracking.

Align on the Security Initiatives Doc

According to Esha, this step is the secret weapon that helps make the whole process so effective. The doc includes:

• Executive Summary: Focuses on the good as well as the bad. Threats, paved road adoption, open vulns, and an overview on the strategic work they should kick off first to mitigate the most risk.

• Partnership Details: Provides context on when the meetings will be as well as the dev point of contact.

• Team Details: Summary of the team, what they do, who to reach out too, documentation/architecture diagrams, and a list of known applications/libraries.

• Security Initiatives Matrix: A prioritized list of security work (from all security teams). They set a high level objective for each goal and define its impact to Netflix.

  • They’ll provide the specific steps to reach that goal and include the priority, description, owner, status, timeline for delivery, and an option for tracking the work in Jira.

During the meeting, security emphasizes what is the goal? It’s not just about fixing bugs, it’s about setting a long term strategic path to reduce risk.

On-going Syncs

During these syncs, the security team’s goals are to build trust and show value, as well as talk about upcoming work and projects the dev team has on their plate.

The security team member ensures work is getting prioritized quarter over quarter and helps the dev team get connected with the right team if they’re hitting roadblocks when working on the security asks.

Scaling Partnerships - Security Brain

Security brain is the customer facing version of all of the security tooling. It presents dev teams, in a single view, the risk automatically assigned to each app, the vulns currently open, and the most impactful security controls/best practices that should be implemented.

While other security tools present a broader variety of information and more detail, Security Brain is purposefully focused on just the biggest “need to know” items that dev teams should care about right now.

Risks With Our Security Approach

Netflix’s approach isn’t perfect:

• There’s a heavy emphasis on the paved road, but not all apps use it, so they have limited visibility there.

• The current automated risk scoring metrics are a bit arbitrary and could be improved.

• They don’t yet have an easy way to push out notifications if there’s a new control they want existing partner teams to adopt.

Evolving AppSec at Netflix: In Progress

Asset Inventory

This includes not just code artifacts, but AWS accounts, IAM, load balancer info, and anything else related to an app, as well ownership and team information.

The Netflix AppSec team is working on creating an authoritative application inventory, which will enable them to better measure security paved road adoption, improve their self-service model and validation of controls, and find where they don’t have visibility.

Prism

Prism builds on top of the asset inventory as a paved road measurement and validation risk scoring system. It will give them the ability to recommend, validate, and measure paved road adoption practices prioritized by application risks and needs.

Prism will enable the security team to quickly ask questions like, “OK, this team has some apps that acces PII. Do they have any apps without logging? Show me all of their apps written in Java.”

Benefits of Prism include:

• Faster IR response and triage time (can determine who owns an app quicker)

• More mature risk calculation and scoring

• Assist in scaling partnerships by increasing self servicing

Future Plans

Over time the Netflix AppSec team wants to grow investment in Secure by Default (Paved Road) efforts, as they tend to be high leverage, high impact, and excellent for devs - devs get a lot of value for free.

Not all security controls can be automated, so making self-service security easier to use is also valuable.

Security partnerships will always be valuable, as there are aspects and context that secure defaults and self-service tooling will never be able to handle. As more of the security team’s job is handled by widespread baseline security control adoption and self-service tooling, they’ll be able to provide even more value in their partnerships.

Stats: This Approach Works

The Netflix AppSec team reviews all of the critical risk vulns they had over the past 3 years and this is what they found:

• 20% could have lowered their risk to High if paved road authentication controls had been adopted.

• 12% could have been prevented or detected with third-party vulnerability scanning (See Aladdin Almubayed’s BlackHat 2019 talk).

• 32% would not have been found with automation or self-service, so security partnerships are an important tool to reduce risk.

They also found they used only 33% of their projected bug bounty spend, which they had set aside based on industry standards, so it appears that they are headed in the right direction.

Maturity Framework for Partnerships

Quick Wins

Determine your application risk scoring model
How do you determine which apps are high vs low risk? Consider using factors like if they’re Internet facing, the sensitivity of the data they interact with, programming language used, and compliance requirements.

Identify teams/orgs to partner with
Consider policies and compliance requirements, business criticality, etc.

Create an application inventory
Automate as much of it as possible so that you’re not constantly maintaining it.

Then, leverage this info in kicking off partnership discussions, consolidate and prioritize the security asks for dev teams, and create an easy to read and track security initiatives doc.

During your ongoing syncs with teams, ask, “What can we do for you?” and “How can we help you?”

Key Takeaways

Use tooling and automation to make data informed (but not wholly driven) decisions. Leverage the understanding you have about the app team’s ecosystem, their world, historical challenges they’ve had, and your knowledge of the relevant business context.

Give dev teams a single security point of contact to make communicating with them and answering their questions easier and less frustrating for them. This in turn helps build a long term trust based relationship with the partnering teams.