- tl;dr sec
- Posts
- Working with Developers for Fun and Progress
Working with Developers for Fun and Progress
Clint Gibler
January 14, 2020
Influential Resources
Leif listed several talks that were influential to him and Segment in building a security program:
Twubhubbook: Like an AppSec Program, but for Startups by Neil Matatall/Brent Johnson (GitHub)
We Come Bearing Gifts: Enabling Product Security with Culture and Cloud by Astha Singhal/Patrick Thomas (Netflix)
Starting an AppSec Program: An Honest Retrospective by John Melton (NetSuite)
Pushing Left, Like a Boss by Tanya Janca (Microsoft)
Leif also recommends the book Agile Application Security: Enabling Security in a Continuous Delivery Pipeline by Laura Bell et al.
Favorite Quotes
“Effective Security teams should measure themselves by what they enable, not by what they block”
Enable, Don’t Block Ideally the security team should say no as infrequently as possible. When you have to, say “No, but…” and help your colleagues find a way to accomplish their jobs.
“Vulnerabilities are bugs. The more bugs in your code, the more vulnerabilities”
Security Has to Start with Quality Devs fix bugs all the time. Vulnerabilities are just bugs that have a security impact. They should be priotized and fixed like the rest.
“Learn to lean on your tools. But depend on your people to keep you out of trouble.”
Choose People over Tools People making good decisions earlier in the process are going to provide more value. You won’t have to find certain classes of vulnerabilities if you make good framework decisions.
“Make it easy for engineers to write secure code and you’ll get secure code.”
Check out Morgan Roman’s DevSecCon Seattle 2019 talk Ban Footguns: How to standardize how devs use dangerous aspects of your framework for one of my favorite talks in this area.
Organizational Buy In
You need the whole company needs to care about security - you can’t just buy a security team and security products. It helps to have a passionate security executive who can translate what the security team is working on to executives.
Security headcount, including people with a range of skillsets, as well as engineering time, to fix bugs as as build scalable systems are also important.
Jonathan Marcil’s Threat Modeling Toolkit informs how they do threat modeling.
If you don’t have security buy-in from the top down I would probably look for another job because it’s definitely an employee’s market right now.
Building a Team
They’ve had success building security teams by hosting security meet-ups, speaking and attending conferences, sponsoring meet-ups, volunteering for OWASP, etc.
Getting involved in OSS software is also great.
Check out David Scrobonia’s talk (Leif’s colleague): Usable Security Tooling, in which he describes the heads-up display he built for ZAP.
Segment’s Head of Security, Coleen Coolidge, also gave a related talk: How to Build a Security Team and Program.
Training
All devs who join Segment go through two security trainings.
Part 1: Think Like an Attacker - Creating Relevant Content
Segment’s security training is interesting for developers because the security team makes it as relevant as possible.
The bug examples devs see in the training are sourced from real Segment bugs, found via bug bounty submissions, pen tests, or internal testing. This makes the vulnerabilities seem real, rather than abstract generic examples.
They also introduce Bugcrowd’s Vulnerability Rating Taxonomy (VRT) to show devs how the security team thinks about the relative severity and priority of vulnerabilities. This gives devs an anchoring in the future for how quickly they need to fix certain bug classes.
OWASP Juice Shop is used as the vulnerable app as it uses a similar tech stack to what Segment has internally.
Hands-On Training Schedule
The security team talks about a few vulnerability classes (slides + examples), then everyone does some interactive training (Burp Suite + Juice Shop). Repeat for other bug classes.
As the training is progressing, the names of devs are being written on a whiteboard as they successfully find and exploit bugs. It’s a leaderboard in which they’re competing against their fellow incoming dev cohort. This really helps with engagement, both by making it fun as well as making people not want to be the only people not on the board.
The Segment security team manages a 1337erboard that tracks people’s security ‘score’. Points are given for attending and participating in security training, finding bugs, fixing bugs, posting an interesting article in the #security Slack channel, etc.
By both presenting information as well as having devs actively engage with hands-on exercises, devs are more likely to absorb and retain the info.
I hear and I forget. I see and I remember. I do and I understand
Part 2: Secure Code Review
Segment bases their secure code review training on the OWASP Secure Coding Cheat Sheet, tailored to Segment’s environment.
The main topics they cover include: XSS, broken access control, secrets management, error handling, SSRF + DNS rebinding, and more.
Leif and his colleague David discussed SSRF and DNS rebinding on the Absolute AppSec podcast #42, including a useful open source security tool Segment released: netsec, a proxy to block SSRF attacks targeting internal IPs.
One of the Segment devs created a web app where you can enter a few keywords and it’ll bring up information from the relevant secure coding guideline
Benefits of AppSec Training
The security team gets to meet new engineers early in their time at Segment and start building a positive relationship
Devs are introduced to common vuln types
“Security Judgment” - give devs a feel for when they should ask for help or do more research when building a feature
Teach devs to apply a security mindset when reviewing PRs
Have fun during onboarding!
Devs get "I Hacked" and "Hackerman" stickers after they complete the training
Vendor Adoption
How do you buy security tooling for devs and have them actually use it?
The Segment security team doesn’t actually buy many tools for their own uses, it’s more often for devs. The security team partners with engineering during the evaluation process to make sure the tooling is easy to use and meets devs’ needs.
Example - Snyk
Leif walks through how Segment adopted Snyk, a tool to help companies manage vulnerabilities in their dependencies. The process had the following steps:
Security Evaluation
First, the security team tested Snyk on various repos, ensuring it had the required features (e.g. language support).
Partner with Engineering
They partnered early in the process with engineering, as devs are the ones who are going to have to use the tool, not security. They did not buy it without seeing if the tool would work for devs.
Presented at Engineering All-Hands
They made it fun by having people write down on a piece of paper how many total dependencies they thought Segment had across all projects and had a Price is Right-style competition in which the winner got a crown.
Initial Roll-out by Security
The security team submitted PRs integrating Snyk to Segment’s core repos. Devs could then add Snyk to other repos based on these examples.
The security team also added Snyk to the Segment starter templates for new Go repos, so any new Go repos created using the standard development process will use Snyk without the dev having to do anything.
Hitch Security to the Developer Productivity Bandwagon
Integrating security controls and tooling into standard dev tooling is one of the most effective ways to get broad adoption of the things you build as a security team. I’ve seen this time and time again and various companies. This also forces you to keep dev UX in mind, as you probably won’t be allowed to introduce a tool or process with too much friction into standard dev workflows.
See Lessons Learned from the DevSecOps Trenches and Netflix’s Layered Approach to Reducing Risk of Credential Compromise for more examples and thoughts on this.
The security team also wrote Snyk integration with Directory, an internal Segment uses to display various useful meta info about services
Some people on Twitter talk about the “security basics,” and I don’t think that’s a good name for it because “security basics” are actually really hard.
“Fundamentals” is a better name. Shooting in basketball is a “fundamental,” yet it takes years and years to get as good as Steph Curry.
Bug Bounty
Pay for anything that gives value. If there’s a duplicate that still gives you value, pay that person for it.
There’s a lot of work that goes into assessing a new target, reading the program scope, understanding the product, etc. If you can build a relationship with security researchers who are going to provide you value in the future, a few hundreds dollars is a small cost to your org.
We’ve gotten all of our most critical findings from researchers we previously paid for dupes or something early.
If you’re thinking about money on a per bug basis, you’re probably thinking about bug bounty in the wrong way.
What other tool do you judge based on how much each individual instance provides value? You should be thinking about the bug bounty program as a whole based on how much you’re spending vs other security tools.
Bug Report → Jira
You want to make bugs easy for devs to fix, so you need a good description, easy to follow reproduction steps, severity, remediation criteria, and a suggested remediation.
If your write-up doesn’t describe the impact in terms devs can understand without help, they’re probably not going to resolve it with the same urgency as you would.
Security ➡ Engineering Embed Program
Segment has found a lot of value in embedding security engineers in engineering teams for a quarter or so. These security engineers follow all the normal software design processes that devs do: writing software design docs, getting buy-in for the feature, working with the design time, writing good test cases, and following deployment procedures.
Ideally the security engineer sits with the dev team, goes through their onboarding process, and basically just part of their team.
The process is a great way to build empathy for devs. Maybe you’ll find the security review process sucks and you can make it more usable.
Benefits By walking a mile in a developer’s shoes, you will:
Make valuable connections- you’ll meet devs, designers, product managers, and other people you wouldn’t normally meet.
Develop a deeper understanding of the engineering process, including their tooling and constraints.
Build rapport with engineering, as you demonstrate that you can build useful features and tools.
Learn more about the code base you’re protecting.
You can then bring these insights and lessons learned back to the security team, helping the team be more effective over time.
Developer Friendly SAST
Leif recommends checking out the LASCON talk by the Datadog security team Orchestrating Security Tools with AWS Step Functions, the Absolute AppSec podcast #33 with John Melton on linting and other lightweight security checks, and Coinbase’s Salus tool.
Check out my 2019 ShellCon talk Rolling Your Own: How to Write Custom, Lightweight Static Analysis Tools for how to use open source tools to create custom checks for interesting code patterns, for example, to detect bugs or anti-patterns.
In Case of Emergency
Ideally you never have to use these, but if you really need to get something fixed:
Compliance requirements (GDPR, ISO27001, etc.)
Recent pen tests (shown to customers)
Customer security questionnaires
“My peers at companies x, y, an z do thing”