Gusto security engineer Nathan Yee describes his experiences and lessons learned on how to be effective as the only AppSec engineer at a start-up, without a senior title.
I liked this talk because it discusses common challenges that resonate with many security teams:
- Given a preciously finite amount of security engineer time, how do you scale the security team’s visibility and impact in a company?
- How do you effect organizational change when you can’t lean on a director-level title?
Nathan was the first security hire at Gusto, and for awhile, the only security team member.
- He was just an independent contributor (IC) without a senior title, so he had little power to officially “force” people to do anything.
- He was outnumbered by developers 100 or 150 to 1 (this is pretty common in my experience).
- As it was a high-growth start-up, the company’s priorities and goals were constantly changing, which he needed to keep up with.
Nathan was fundamentally limited by his finite amount of time, so the question he asked himself was:
How do you scale yourself across developers and non developers alike?
Nathan discusses two core types of strategies: technical and cultural, and concludes with two examples of nice security wins that happened because of the positive security culture he helped build.
The technicial strategies he described are pretty standard: “shifting left” with continuous static analysis (brakeman), scanning for vulnerable dependencies, penetration testing, and bug bounty.
Here are some of the unique tidbits/comments:
- Modern security teams are significantly higher leveraged when they can write code, calling out Dino Dai Zovi’s 2019 Black Hat USA keynote.
- Another huge value of being able to code, which I think is often underappreciated, is being
able to speak the language engineers use every day is incredibly powerful.
You can work together as partners on the “same side” with a common vocabularly.
- It gives you empathy into what developers’ lives and perspectives are, enabling you to better align your needs and goals witih their workflow.
- Being able to build security features for engineers or push code that fixes a security bug also earns you their respect- you’re committing code too, not just pointing out flaws.
- Nathan created a custom check that will send him a Slack message automatically
whenever any PR touches a
brakeman.ymlfile, which contains the set of bugs whitelisted by
brakeman. This gives him continuous visibility into how scan results are being addressed over every repo without having to manually examine every commit.
- One of the core values of lightweight scanning tools like
brakemanor linting tools, beyond the issues they identify, is that they give developers fast feedback. Giving timely results is critical (vs sending them results about code they wrote days ago, where they’ve lost the mental context).
brakemanscanning every commit for common bug classes, developers can write and push code at their own speed, without being blocked on waiting for an AppSec engineer’s time.
- Review your pen test reports over time as a gauge of how you’re doing; hopefully the number and severity of findings is trending down over time.
Build Relationships with Key Stakeholders
Including IT, compliance, product managers, legal, engineering, and infrastructure.
- IT can be the front line of security, e.g. “I clicked on a sketchy link and now my browser is acting weird.”
- Befriending product managers is key because it helps you insert security into
feature planning itself. Also, when they trust you, they’ll proactively inform you
when there are upcoming plans to touch security-relevant functionality, such
as authentication, authorization, etc.
- These inbound heads-up are incredibly useful, because the security team can’t join every sprint planning meeting or really even be aware of what all of the different dev teams are working on.
Be Open and Vulnerable
Share the results of bug bounty submissions and pen test reports with developers. Show them your successes, be open about your challenges and failures.
This openness will build trust with developers and make them want to help you.
When Nathan was the only security team member, he made an effort to eat lunch with a different team every day. That helped him, and the team members, put faces to names and start building rapport.
Preemptively Fill Up the Security “Trust Bank”
Take advantage of opportunities, however small, to help out, provide value, and build trust with developers and other parts of your company before you need it.
Once Nathan took the time to help a developer who was having trouble with 2FA on a personal (non-Gusto) account. This wasn’t work-related, nor would it fall under the security team’s purview if it was, but Nathan took the time to help him figure it out.
This freely offered help, with no intent of a corresponding favor, ended up generating good will that was crucial several months later when Nathan had a big security-related ask of this person.
“You’re going to need to make security withdrawals from various accounts at some point; put in the upfront deposits to make those as seamless as possible.”
An hour of teaching web security to 15 devs takes one hour of his time, but then they go off to their respective teams with security on the top of their minds.
A benefit of training, beyond the knowledge shared, is that it allows Nathan to put a face to names for devs and makes them aware that security is a friendly team within the company that they can reach out to.
Wins from a Positive Security Culture
In one case, Nathan had an engineer proactively reach out, saying that a certain group appeared to be overprileged.
Since engineers are in the trenches day-to-day building the software your company runs, they generally have the best context of how systems work in detail as well as what level of access users and groups should need to perform their required tasks. Being able to lock down this group was a great win.
In another example, a developer proactively reviewed HackerOne reports from the night before, determined that they were real issues, and then submitted PRs fixing them about an hour later.
Nathan didn’t have to badger anyone to review HackerOne repors or convince them of the value of fixing the bugs, they proactively went from triaging through fixing the vulnerabilitiess all on their own.
“Relying on a CSO or Director title to influence is a problem that positional power doesn’t solve. True power comes from a desire to understand and solve challenges with software and relationships.”