• tl;dr sec
  • Posts
  • (Transcript) Lessons Learned from the DevSecOps Trenches

(Transcript) Lessons Learned from the DevSecOps Trenches

In a sentence: The transcript for this AppSec Cali 2019 DevSecOps panel featuring security leaders from Netflix, Dropbox, Datadog, Snap, and DocuSign.

Clint Gibler, Research Director, NCC Group twitter, linkedin
Dev Akhawe, Director of Security Engineering, Dropbox twitter, linkedin
Doug DePerry, Director of Product Security, Datadog twitter, linkedin
Divya Dwarakanath, Security Engineering Manager, Snap twitter, linkedin
John Heasman, Deputy CISO, DocuSign linkedin
Astha Singhal, AppSec Engineering Manager, Netflix AppSec Cali, Santa Monica, twitter, linkedin CA. January 25th, 2019
🔖 summary 💬 abstract 📹️ video

You can read the summary here.

Panel Overview

Clint Gibler 0:08
So I think we can get started. First off, thank you very much for coming. We’re very excited and honored to be here. We don’t have any slides, this is just going to be an informal discussion amongst people who have all done some pretty neat security automation things in different companies.

So we’re going to talk about some of the sort of tips and tricks they’ve learned over time. Some things they tried, that didn’t work as well. And we’re also going to allow a bunch of time for questions. So please, feel free to ask a bunch of questions, even start thinking about them now. Because if no one asks any questions, we’re just going to sort of awkwardly sit here staring at each other for perhaps 20 minutes.

Also, just to give you some additional context, we all chatted before this, and we all agreed that panels where everyone agrees about everything are very boring. So we’re purposely going to hopefully find some things we don’t agree on. And we’re all friends. None of us take it personally. But we think that that’s interesting in that the same idea or tool or process may be very useful in one company, but less so in another because of culture or other things. So hopefully, we’re going to tease out some differences here.

Okay, with that being said, let’s just quickly go through everyone and just say, your name where you work. And yeah, just to introduce yourself.


Devdatta Akhawe 1:25
All right, hey, I’m Dev, I work at Dropbox in security. And yeah, is there anything fun factor and-

Clint Gibler 1:34
feel free to do a fun fact, or like a sentence about maybe how the security teams are structured within your company?

Devdatta Akhawe 1:40
Okay. Yeah. So I lead this team called Production Security, we’re just focused on basically all aspects of protecting our users and the data. And so application security, abuse prevention, abuse and threat intelligence, and data security. So pretty broadly anything to do with protecting our users and our product platform. Some somehow is in this team. So yeah. Hi, everyone.

Astha Singhal 2:06
I’m Astha, I lead the app sec team at Netflix. And so my team is sort of responsible for securing all the applications that help run the Netflix business, as well as the Netflix streaming product. And then we also run the bug bounty program and piece or capabilities for Netflix. And then I’m part of the broader team that does sort of engineering security, which is sort of within our broader info sec organization.

John Heasman 2:34
Hi, everyone. My name is John, the horseman. (Editor’s note: this auto translation was too funny, I had to keep it.). I’m the Deputy CIO at DocuSign. And I run a security engineering team that’s application security, vulnerability management and security infrastructure. And my fun fact is that when I started the app sec team, I was actually an engineering. So I was in engineering for at least half of its life at DocuSign. Now, we’re in info sec, interesting pros and cons as to how you structure your teams.

Divya Dwarakanath 3:00
Hi, I’m Divya, I work on the application security team at Snap, I manage that team. Our team works on building secure default frameworks and tools, automation tools, the security reviews of new products and services that we ship. We also do reviews for services that already exist that we are nervous about. We run the bug bounty program. And we also contribute to the largest security training efforts at Snap.

Doug DePerry 3:39
Hi, my name is Doug DePerry, I’m the director of products almost forgot what I did per second. I’m the Director of Product security at Datadog. I realized that Datadogo is not as much of a household name as some of these companies. But Datadog is a cloud native SaaS monitoring solution. We ingest metrics and logs, application traces and provide to customers in a single pane of glass. And what my team does is focuses on, you know, application security, kind of three main tenants we we build software to kind of help developers do their job more securely, we break software to find bugs and do code reviews, that sort of stuff. And then education is kind of that last piece. And so that’s kind of our approach to application security.

Clint Gibler 4:23
Awesome, cool. Oh, and my name is Clint Gibler. I’m a security consultant and research director at NCC Group, we do penetration testing, and pretty much anything related to security. And I started thinking about this in a lot more detail once I was working with a number of different companies, helping them set up security automation, helping them figure out where static analysis can give them the most value or not, and things like that. So I’ve also been very interested in this space. Okay, awesome.

SDLC / Security Automation

Clint Gibler 5:00
Let’s maybe go down the road and talk a little bit about can you give us a high level overview of how your SDLC works, and sort of what security automation is current in place. And if you want maybe one or two things that you have found to be the biggest ROI from this?

Doug DePerry 5:11
So Datadog has a pretty informal SDLC, we don’t put a lot of, you know, we don’t do a lot of ground rules as far what language they can use, or what frameworks and that sort of stuff. And so, you know, it’s a lot of freedom. There’s a lot of dev teams, that’s a lot of things for us to kind of keep track of.

And so we automate, one of the main things that we’ve done lately is, is we automate just trying to keep visibility, we call us ProdSecOps. And so we kind of have a, a central place where we can submit stuff, like, you know, we monitor a very specific source code files in GitHub, for changes, stuff related to like author or you know, anything sensitive like that.

Trello cards that have a security label, we send the results of some of our scanning systems to this thing as well. And that’s, we call it ProdSecOps. And it’s, you know, every team member takes a one week turn, and they kind of just, you know, just really helps us gain visibility on what the org is doing without having to be everywhere at once we can, you know, attend, you know, attend every single stand up, because not every team has a standard every day and stuff like that. So that’s one of the main ways that automation has been providing a lot of value for us.

Divya Dwarakanath 6:20
Yeah, it’s a lot similar as the airlock at Snap, we use a lot of different frameworks, languages, a lot of ways to do things. There, there’s a bunch of automation tools that we’ve hooked up to different stages of the development lifecycle, lifecycle thing called that we have pre-receive hooks that sort of look for certain things as you push code out, to GitHub. We have analyzers that look at your code, as you open your pull request and like comment on them would like hey, you should be using some other tool instead. We have tools that we run post deploy, looking for, like certain issues, like, Did you misconfigure auth, should this be public or private, etc.

And we also have tools that we want people to use, like, this is a key manager that you should be using to do to store your secrets. And just a secure by default framework, this is what you should should be using if you want to do X.

John Heasman 7:32
So Clint told me I had to disagree with everyone. So we don’t do any of that.

No, I just want to highlight a couple of things we do. Starting with training, we wrote all of our AppSec training in the house, it was a hugely time consuming exercise, but we looked at sort of commercial offerings. And we figured it’s rare that developers at DocuSign, start building an app completely from scratch. They’re using our existing frameworks, our existing components, and they need to how to use those securely. So really, our training focused around using using those.

All throughout our SDLC, we tried to have lots of touch points with developers, and we try to keep it sort of pretty lightweight. So one of the things we do is we have team set up in all of our orgs in GitHub, so that any point developers on a PR they can they can mention the app sec team. So it’s like a nice informal way that developers can actually ask us questions. Hey, @appsec, like, you know, is this line correct? Or should we be thinking about this? So really, yeah, our process was: “Think of all the ways throughout the SDLC or the phases where we can just have a low friction way of getting that visibility.”

Astha Singhal 8:43
So yeah, for us, we have sort of like enough infrastructure, we sort of have this concept of the Paved Road, which is sort of like you can go do whatever you want be able to a polyglot environment. But then there is like the, you know, paved road for supported things. And then we have a similar concept for security.

And then in general, like, what we focus on a lot is how can we get folks to adopt the security paved road. A lot of the tooling that we’ve built historically has been sort of in the space of things that we consume ourselves. And we tried to sort of expose it to the developer, once we’ve had a chance to like verify an issue or something like that, because we have a strong bias towards not interrupting the developer if we don’t have to.

And then we also have some automation in sort of like the, like the self service from the standpoint of you can submit a security questionnaire to get a standard set of guidance based on your use case and things like that. But then, sort of looking forward in 2019, we’re sort of leaning heavily into the Self Service space from the standpoint of how can I tell you what to do for security by looking at everything about the about your app, and your app inventory, and your risk based on how your app is deployed, and things like that. So that’s kind of the major sort of automation we’re doing.

Devdatta Akhawe 10:02
Alright, so Dropbox has the best…

We think about security throughout the SDLC, automation throughout the SDLC. So design phase, implementation phase, launch. So during design phase, there’s involvement from security, we strongly encourage developers and evangelize to them that they reach out to security during design phase.

There’s design reviews, threat modeling documents, standard frameworks on how to design things right. And then there is the implementation phase, there’s automatic code review blocks, automatic code audits based on static analysis that flags suspicious code.

One of the cool things we do, as product teams roll out new features, to early access, we actually roll it out to we have a special population in our A/B testing framework, that is our bug bounty crowd. And we actually automatically roll it out earlier to the bug bounty crowd saying, hey, if you find a bug, before it rolls out to the rest of our users, that’s even better for us.

And then finally, there’s dynamic analysis and continuous testing after the feature has been released to everyone. And so, you know, developer relationships throughout.

And there’s a whole separate focus on platforms where we look at what are the common bugs that are affecting people throughout this lifecycle, and where we can write better libraries, better developer education, better dynamic analysis, static analysis to detect and prevent these flaws. And so a lot of it is I agree, developer relationships, one of the things, I think that was left unsaid that we have also found very powerful as relationships with product managers, because, you know, they are the ones who are thinking about the product, and what we want to be the earliest, often before engineers, and so working with product managers and getting involved during their design and ideation phase, and evangelizing how we think about security has also been pretty powerful. So yeah.

Continuous Code Scanning

Clint Gibler 11:49
One interesting trend I found, talking with you all, as well as talking with a number of other companies is that so many people have basically built the same sort of internal tooling that, you know, given a new PR or commit, run arbitrary pluggable tools on them.

So I think everyone on this panel, have done that at their company. And I’m just curious, like, quick poll of the room, who is at a company who’s like build something for your use cases that automatically scans either static or dynamically new code pushes? Like, is this a pretty common thing? Yeah, tons of people. Is this something you’ve written yourself? Or is it like a tool you bought? Or I guess, raise your hand? If it’s something you wrote internally?

Okay, most people. Has anyone bought a thing?

Devdatta Akhawe 12:40
Are you saying bought and used or bought and stopped using? (laughter)

Clint Gibler 12:47
Yeah, did you did you buy something and then it just didn’t fulfill your needs, because of specific things, like how Dropbox works?

Devdatta Akhawe 12:54
I don’t know whether it’s how Dropbox works. I do think it’s a very, I’ll talk specifically about code blocks. It’s a very, I don’t know what to call it like a very rude thing to do to developers telling them their code is not good. And so something that we can write and manage in house, update in house has been pretty powerful.

We bought in the sense that we asked our, so we use Phabricator internally, we asked Phabricator to update their tooling, so that one of the very powerful things that we did was, developers can upload a new diff. And the static analysis runs and adds a blocking security reviewer and leaves a comment saying, “This is why the blocking security reviewer was added.” And if they fix that thing, the reviewer automatically goes away at the next diff.

And that was a feature that was actually uncommon in many of code review systems. But just that has been such a huge win. Developers love the ownership they get, and and also reduces overhead on us. So that’s been pretty powerful.

But in general, yeah, the tooling we use for this is in house. Some of this might be Dropbox specific, because we tend to focus on secure libraries. And so you know, we just say like use the secure version of crypto, we just automatically block on using any crypto function that is not the the one we have blessed. And so that’s a very simple stat. Like, I don’t even know, that’s a simple grep, right. So, so that worked better than many of the other engines that reliability and speed and all these issues were out of our control.

Divya Dwarakanath 14:17
Yeah, we did something similar with the the thing that comments on diffs. It’s not a blocker, but it gives the devs the opportunity to be like, this doesn’t make sense. It’s a false positive. Let me thumbs that down. And then a security engineer can go look at why was this a false positive? And how can I improve tooling around that?

Devdatta Akhawe 14:35
You track metrics around number of thumbs down, thumbs up?

Divya Dwarakanath 14:39
Do we track metrics? Yes.

Devdatta Akhawe 14:41
Oh, that’s really cool.

Clint Gibler 14:44
Yeah, and have you actually use those metrics to inform, like, customize the checks over time? So like, “Oh, this rule tends to be like, 90% false positive, so let’s either make it more precise, or maybe remove it entirely.”

Divya Dwarakanath 14:56
We’ve done a little bit, not a lot, but it’s an ongoing process.

Clint Gibler 14:57
Okay, nice. Very cool.

Devdatta Akhawe 15:00
Yeah, we tend to look at, you know, resource vs reward. And if there are particular tools that fire a lot, and they are not very high risk, we kind of move them to audit, which is post commit, rather than during the code review. So this allows developers to move faster, and then we can later on go in and be like, “Okay, you’ve committed the code, this is not urgent, this is not the worst thing in the world. But just this is not hygienic, like please change it, please.” And you’ve dealt with like that.

John Heasman 15:25
So I think one thing we probably all be in agreement in is, obviously static analysis, has the potential to flood you with false positives. One way to limit that is essentially changing, changing the problem to as Dev said, enforcement of secure libraries.

So at DocuSign, we wrote a bunch of wrapper classes around potentially dangerous operations. And now we enforce their usage over the original framework version. And that’s a super simple problem for static analysis to solve.

But one thing I wanted to say on that is, you know, how we persuade all of engineering that, “Hey, you got to use that thing. It’s better to use our thing, our our wrapper code”.

The way we did that, actually, we canvassed developers, and we found that one big thing at DocuSign is telemetry driven design. And so we built telemetry into all of these components. So let’s say it’s a component to prevent server-side request forgery. So it’s a wrapper around like, you know, something that makes an HTTP request. Well, we also baked in a bunch of telemetry. So you know, we can easily say to our engineers, hey, use our component, you get this for free, you get this monitoring for free.

Astha Singhal 16:37
For us, a lot of the code analysis stuff is actually more sort of greppy trying to find anti patterns. And that has worked better for us than actual static analysis. Just because I think we’ve done sort of some experiments and testing, trying to run static analysis on large parts of our code base, and at the end of the day, like that the risk versus reward just hasn’t been worth it for us. So we’ve leaned more into kind of like finding anti patterns, as opposed to doing sort of like full static analysis.

Divya Dwarakanath 17:10
This is a question for you. Sorry, I’m asking the questions,

Clint Gibler 17:13
Please do, I can just hang out.

Secure Wrapper Libraries

Divya Dwarakanath 17:19
How difficult was it for you to drive adoption of these wrappers because it’s one thing to build a wrapper library, and another to enforce it and to have thousands of applications adopted.

John Heasman 17:33
So I think, like a lot of things that we roll out, we don’t go from zero to 100%. So we don’t go from no one using it to everyone using it within a week. This goes for a lot of things- partner with teams that want to use this stuff, that want to be good partners with security, roll it out, make sure you iron out all of the bugs so that when you present it to teams that might be slightly more hesitant to adopt it, you can be pretty guaranteed that we’re not going to run into problems. So I’d say that was probably the easiest way.

Devdatta Akhawe 18:05
We in our case, we just went in and did the changes ourselves. It was painful. But it felt right. Like why should we export our pain to all the other developers team. And that taught us a lot, right? Like, there have been cases where we thought the secure design we were proposing was this beautiful thing, and it’s secure. And then when you try to actually use it yourself, you’re like, Oh, this is a terrible idea. And so then you do a better design. So part of that is also like forcing yourself to use the APIs you’re providing kinda teaches you a lot and makes you rethink what’s appropriate.

Doug DePerry 18:40
I was actually thinking about, you know, kind of the earlier question a little bit about, you know, why couldn’t you use, like, why’d you’d have to build this yourself sort of thing. And I just, we find a lot of times, and it’s not that Datadog is doing stuff, that’s, you know, so off the wall, like, we use GitHub, you know, it’s not, you know, there’s a centralized place to do a lot of this stuff.

But I just think the way some of these tools are architected, and I’m not going to pick on static analysis, but really static analysis, it just, you know, we want to, I don’t want to get alerted for every single problem, or my team, I don’t want developers to have to log into a separate web UI and learn a whole separate way of doing things, I want to present it to them in a way in which I know will work for Datadog’s development culture in with their development workflows, because I don’t want to make things more difficult for people. And so that means I’m going to have to do things I can’t expect every product to fit into that.

And so we end up writing a lot of custom code and wrappers around, you know, tools APIs, which has worked out pretty well for us too, because it’s also it’s more extensible. And we can kind of plug in, you know, additional tools as time goes on. But it’s, you know, it’s a model with a lot of tools. And it’s not just static analysis, it’s not just log into our thing, and well, no, I don’t want developers to do that, I don’t want to have to deal with all that, I want to make it I want to present to them the problem in very simple terms, and show how they can fix it very quickly and easily. And if they have problems they reach out to to the security team, not hitting up support for you know, this, this web portal or something like them.

Clint Gibler 20:16
Yeah, and just to parrot something John told me like a while ago, but one of the reasons why they built the orchestration part is that by you being in control of the glue, and fitting things together, you can easily plug in a bunch of tools, where if you’re relying on a vendor to do that orchestration and glue, you’re sort of dependent on them. But by owning that part, you can sort of plug and play different tools based on if they’re useful or not to you. And I thought that was a very insightful way to do it.

Devdatta Akhawe 20:42
So I come from an academic background. And so I have a PhD in CS. So I am very hurt. All the shit talking of static analysis was hurting

Clint Gibler 20:53
for the record. I love static analysis, but yeah, but I also agree.

Doug DePerry 20:57
Yeah. We all love. Yeah, all love grep, it’s great.

Devdatta Akhawe 21:02
But, but one of the things that actually, you know, I was doing some static analysis research, even in grad school, and one of the things that someone told me I forget who, was the reason why grep is so nice is because, because if you think about it, grep is wrong, probably more than static analysis, because static analysis has some intelligence in it. And so grep is usually wrong much more often.

The reason it is less annoying to developers, though, is because developers understand grep. So very often, they see why grep went wrong. And they might even tell you, hey, you need to fix your regex, and really enjoy telling the security team how to fix their regex. And, and that’s, that gives them a very positive like, rush.

I mean, yeah, regexes are hard, dude.

But, but that’s fine, right? Like, in the end, the relationship is important. And annoying developers less is very important. And so you know, this magical black box said, “You have a bug” is super annoying to developers. Grep has a bug an ddevelopers know why it’s wrong, is actually even though it’s wrong more often, is less annoying in my experience, which is, you know, one of those things that I don’t really like really understand when I was in grad school.

Clint Gibler 22:07
I think there was actually a like a white paper from Coverity while ago that was talking about how they improve their scanner engine to find some very complex, detailed bugs. And it actually wasn’t a good thing, because it was like, “Oh, this is actually a bug”, but it was so confusing to people reviewing it that unless you knew like esoteric see language features, like it would just seem like a bug. So it actually wasn’t useful to them because of that. So being able to understand the results is obviously very important.


Clint Gibler
Okay, so one last thing before we open it up for questions. So many times people are giving conference talks like, “Hey, here’s this cool thing we built” or “Look how awesome our program is.” But I think less often people talk about failures they’ve had or things they’ve tried that didn’t work, which I think is a loss for the community, because it hopefully we can save ourselves a lot of time trying to go down avenues that didn’t actually pan out. So yeah, just what’s something that you’ve tried that didn’t, that you thought would be very useful, but wasn’t, and not something you like, tried to hack on for a day, but something you actually invested, you know, a non trivial amount of resources in that just didn’t either work at all, or pan out how you expected?

John Heasman 23:14
I would say, making certain commercial DAST applications log into a single page app. Has anyone here had a similar problem? Yeah.

It kind of feels like, in particular with DAST, a lot of commercial tools are just behind modern development. And that makes it super, super painful. But we’re obviously told that you need DAST for various compliance reasons. And you know, it makes sense when you think about it. Yeah, I’d love to do dynamic analysis. But so that’s something Yeah, we worked on for a while.

Ultimately, you know, tried a bunch of vendors and still have actually not had great success. The way we eventually solve this and actually got decent coverage is, we went and talked to our QA team who had a great set of functional sort of end-to-end tests. And we basically hacked on their code to add security into those QA tests.

That had some pain along the way, when we started breaking tests, you know, QA were like going crazy, why the hell have you broken this test, but you know, that’s sort of, we managed to repair that damage, and we have some really great coverage. But we still do use a commercial DAST tool, but we like to compare it to results with the results we get from our in house QA system, and as you would imagine, they’re just non comparable.

Doug DePerry 24:44
I think one of our biggest failures, I guess is or it’s actually a little bit entertaining, too is, that ProdSecOps thing I was talking about earlier. We, we were looking into using keywords in Slack, to, you know, we want to try to, like get involved as early as possible, right? We want this visibility throughout the org. And we’re like, “Well, hey, people are talking in Slack about this new feature, this new thing, I bet if we look for words, like you know, “hash”, or “crypto” or “MD5” or something like stuff, like that’ll give us like a little bit of insight into, you know, what’s happening, we can join the conversation. And it’s really cool, because we’re just friends chatting on Slack. And it’s all good.

It turns out, it’s really, really creepy just to pop into public, you know, even though the public chats and it was it didn’t really work out too well for us. So we didn’t put a ton of time into it. But it was still like you always got to think about the creep factor of what you’re doing. Like all this monitoring, all that. all that sort of stuff that telemetry, it’s all great. Just remember that, you know, people are people to you.

Clint Gibler 25:45
Yeah, like ProdSecOps Beetlejuice.

Doug DePerry* 25:45
Exactly. Oh, here we are. What are you guys talking about? Like, how are you in this channel?

Divya Dwarakanath 25:56
So a year or two ago, our company developed a lot of features, lot of products we were constantly shipping. And developers used to ask us for security review, all the time. So we had this huge queue of like security reviews we had to deal with. And our turnaround time was understandably very, very high.

At the same time, Google released this thing called vendor questionnaire (VSAQ). And we were like, someone came up with the brilliant idea of like, Oh, great, why shouldn’t we like, take that, change it? So we asked you a bunch of security questions or just a bunch of questions about, you know, are you building a web app, mobile app, are you using web view, or XYZ? And depending on how they answer like, the tree expands even more, and gives you advice on like, you should be doing X and not Y, you should like look at the documentation, and how to use how to generate randomness, etc.

And we had really high hopes on that.

And turns out, the developers still just wanted a human to do their reviews. So they would just find the shortest path to completion of that questionnaire. So just be like, web, mobile, other submit. So didn’t quite work out. But you’re sort of trying to figure out how to improve on that.

Astha Singhal 27:23
So it’s interesting because Clint told us to disagree with each other. Actually, the question everything has worked pretty well, for us, it has helped us sort of create touch points with teams that we don’t normally will work with. And then like, sometimes they’ll either reach out to us, like on the #securityhelp Slack channel.

But then a lot of times, they will fill out our Zoltar surveys, which is sort of like a similar questionnaire thing for us. And then I think the one thing that didn’t quite work with that is I think we had functionality for actually automatically like filing Jiras for what you were going to do for security work. And I think we’ve had maybe two or three people use it ever. So that part of it was the fail. But the actual guidance piece, I think, like worked in our environment. But anyway, it is a failure story.

I’m telling this next example on behalf of Scott who is on our AppSec team, I think a few years ago, they spent what was it, six to nine months, getting an Internet-facing deployment for a dynamic network scanner. And at the end of it just really didn’t find anything.

Yeah, that’s the end of that story. Ever.

Clint Gibler 28:32
Yeah, and I think, to me, that’s sort of a, I guess it’s too strong to say like an inspiring story. (Astha laughs)

Context– so I feel like, I’m a big fan of the various talks Netflix gives, I think they’ve released a bunch of very cool tools. And many of the talks you give I think, are very insightful and useful in this space, specifically. So I think it’s nice to see like, oh, even people who do such great work in so many contexts, like, you know, no one’s perfect. So to me, that’s why I said, inspiring.

Devdatta Akhawe 29:06
The list of failures is very long. It’s like that moment in the movie when the actor has 100 things to say.

I don’t, I think I just jump off of what Divya said, we also had a very similar idea, we failed for a completely different reason, which is funny.

So we thought yeah, like, we had this threat modeling form. We learned from Adam, we took the like, you know, what are you building? What could go wrong? Like the standard threat modeling form. We took away the I mean, based on initial developer feedback, we took away the “did you do a good job at it” because they got mad about it but but we kept the three questions.

And and then after while we’re like, you know what, it’d be really nice to make this like really beautiful flow. And people can click through when they say “web” we’ll ask these XSS questions. And they say like “desktop” we’ll say these RCE questions like, it’ll be great.

We had this visions of like, you know, UML documents and like these flows and stuff, it was going to be great. Rolled it out, developers hated it. And the reason they hated it, I think maybe you have a nicer team than (to Astha)- I don’t think anyone said “I want to talk to a human” maybe because they have to talk to me. But like, but, but the reason they hated it was funny.

Before this, this was a Dropbox Paper Doc, which is like, think of it like a really nice version of Google Docs. But, the thing is, in a Dropbox Paper doc or a Google Doc, you can collaborate with others in the team, you can work with the product manager, you can work with your designer, your eng manager, tech lead, we can all collaborate together. A question they don’t know the answer on, they can comment and ask each other questions.

In a form or a survey, you can’t do that. I was like, Yeah, that makes sense. There’s a reason why we invented stuff like Google Docs. It’s miserable to write like, I don’t recall the last survey, I felt where I was like, Oh, that was such an amazing experience. I hate surveys. And so. So yeah, we killed that, we went back to a simple form where people could work together, answer each other’s questions and that continues to work much better. So. Yeah.

Audience Questions

Clint Gibler 31:05
Very cool. Alright, so does anyone from the audience have any questions?

Okay, I can go ferry mics around.

Clint Gibler 31:18
Also repeat the question.

Audience Member Okay. Yeah. So any thoughts on storing secrets used by security tools?

Devdatta Akhawe 31:26
I don’t have good advice on, because I think we’ve done it most like, like any of our other code. And the security team has worked very hard to have like solid secrets management infrastructure, in production. And so we use that. And the one thing is like, the credentials we use for testing the accounts we use for testing. If they get hacked, like fine, like we don’t like it’s also not sensitive. Like they’re not we don’t use any sensitive accounts. But also, yeah, we use our normal secrets management infrastructure. But if you don’t have a secret management infrastructure, yeah, I mean, honestly, keep it in the file. Like if you don’t have secrets management infrastructure there are other bigger problems.

Divya Dwarakanath 32:04
like sensitive credentials, and you don’t have a secret management solution we’d consider something like, no, like an s3 bucket, but relying just on like permissions for access, rather than worrying about like, should I keep that encrypted wallet, the bucket becomes public, etc.

Or KMS on AWS, or GCP

really depends on where your tools are running and what is available to them to share. And there’s no one size fits all answer here.

Astha Singhal 32:37
Yeah, we also rely on sort of like our secrets management solution that our platform security team builds. And I think I’d argue that, like, that’s probably a more important thing to build, first than building a bunch of other security automation. So when you have these foundational security services, those are really important part of making your AppSSec program successful, because you need to have a solution, you can tell your developer that they can easily deploy to solve these problems you’re finding, right? Because it’s like, if you don’t have a solution to point them to, then you finding the bugs doesn’t really matter.

John Heasman 33:12
Yeah. And if that solution isn’t well documented, and easy to use, that’s when you end up with, you know, creds in code. So I think no matter the level of maturity of your program, always scanning for secrets in source code, like something will pop up now and again, no matter how mature you are.

Audience Member 33:30
If you’re at a company with a new, small AppSec team, where do you start?

Clint Gibler 33:32
Yeah, that’s a great question. So for a young, blossoming, exec team that you’re at, yeah, it would be nice to have all these things already built. But let’s say you join a new company, small team, where do you start? What are sort of the nice big wins? Or where should you focus your time initially?

Astha Singhal 33:50
I think at that point, I think when you’re a new team, the biggest thing that would matter is like actually building those relationships with parts of the company that are building sort of like the highest risk assets that you care about.

So think through kind of like what that is, for your business model, right? Like, for us, it’s credit card information, and like PII and viewing history, and all of that sort of stuff, right? So you should like go and spend the time on building those relationships with those teams and like, try to really understand the problems they’re facing. And then like, maybe use that to sort of prioritize how you want to start spending your time because they can tell you a lot more about the security problems that exist in your ecosystem.

Doug DePerry 34:32
I was going to go with something like cases of bourbon, but um, it is very overwhelming. And you have to kind of just, you know, brutally prioritize, you know, and I’ve been saying that for years, I’m still not good at it. Right? It’s, it’s difficult because it changes every day, several times a day.

One of the first things I did at Datadog when I started because I was, you know, one of the first handful of security hires was, I focused on infrastructure security, because that’s what we need, like, that was what was going to bite us the worst. I mean, you know, my boss hired me to do application security, product security, and I’m like, I’m going to go do infrastructure stuff.

He said, “Do you know much about it” and I was like, “Well, no, but I’m gonna figure it out because we need to.” And that was, you know, that’s kind of how I prioritize that sort of stuff.

So I think brutal prioritization, and then just kind of keeping up with it and not forgetting about that stuff, either. Make a list and keep it and then you can revisit it when you’ve got five or six people, and you’ve got multiple teams and that sort of thing.

John Heasman 35:26
I would say, seconding, sort of building those relationships, because that’s a great way to actually build out an AppSec team hiring internally. Sure, we all agree, it’s really hard to hire great AppSec people. One way to do that is to find those people that are interested in security and QA and ops in engineering, start with maybe a security champion model, and then eventually persuade them to join your team.

I would also say look across sort of historical data on bugs, you have identify trends, identify systemic vulnerabilities, don’t play whack-a-mole, try and solve that entire class. So that goes back to building like wrapper components, so you can just eliminate entire classes.

Divya Dwarakanath 36:09
So when I joined Snap I was the first AppSec person I think, maybe the second. So yeah, the security program was pretty, like non existing. I think the first thing that, and this may be like baby steps, but you really, really need to get to know the infrastructure and what you’re dealing with. Ask people a bunch of questions. And as you’re talking to them, you realize like, there are some clear gaps that you need to you can immediately start tackling.

And, you know, going off of what the others said, if you’re looking for specific teams that you should build relationships with, like a B2C, it’s probably like a growth or identity teams who are going to run in with into a lot, because in a lot of ways, their goals are quite the opposite of yours. So to make sure you get to know them and stay in touch so that you get all the latest scoop of like, what they’re building, and you can really be an advisor and take care of those teams.

Devdatta Akhawe 37:19
I would just add one more thing. So I agree with everything, unfortunately, because the plan was to disagree.

But I think the other thing I’ve seen many others trip up on and there are also a bunch of good talks at this conference on how to start up a security program. The one thing I’ve seen people trip up on in, like, my friends and other security programs is the choice, right? The analysis paralysis of like, Oh, do I do this? Do I do that. And, you know, it says a lot about how nerdy I am, that my management references are all computer science.

But I think of it as like a query optimization problem. And if you know how databases do query optimization, they set a fixed time to make a decision. And then they just make the decision based on how much like at the end of that time, what they think is best, and then go with it. And then after a while, of course, all query optimizer also see how the queries actually running. And if it’s taking too much time, and it’s taking too much time or is going wrong, they again, re change based on the data that you’re seeing. So so I think of it like that is just like make a choice, set a time make a choice. And then see based on all the things that people said- right relationships input from the business, data classification, maybe the choice you made is wrong, maybe it’s right. But if it’s wrong, change it. But But I think analysis paralysis this is where we just don’t do anything for six months is probably not great either.

Clint Gibler 38:32
Yeah, I think the mentions of building the personal relationships, very interesting. But I guess one thing I wanted to ask is, if you had to choose between, so definitely emphasizing the relationships and the people part, but between, say, building some security automation in terms of like automatically scanning commits and things like that, versus building wrapper libraries. So like of those two, if you had to choose one, which would you do first? I guess you could cop out and say some combination, but try not to.

Devdatta Akhawe 38:59
I think, you know, it was pretty clearly writing wrapper libraries. But again, we are, you know, we do not have like, arbitrary number of different stacks, we have a philosophy of like supported framework, supported stacks, we were a monolith for the longest time. So in that case, it made total sense to focus on that wrapper libraries first. Because almost always in security, preventing a vulnerability is going to be, you know, better than finding it later. So in our case, it made more sense.

I suspect it’s not going to make sense for others. And if no one plans to disagree with me, then I can disagree with myself, but like, let me know.

In some other cases where we have like hundred different types of languages, hundred different types of frameworks, then it wrapper libraries just don’t scale, right. Like one of our team members gave a talk at this conference on internal Application Security where we had this problem. There, we focus more on detection and runtime monitoring. Because yeah, you can’t write rapid libraries for all the different frameworks and languages everyone wants to use. So yeah, that’s the trade-off.

John Heasman 40:01
Okay, now I can disagree with you. Relationships don’t scale, that was going to be my point. If you look at sort of maturity of growing a team, any team really, you might be able to get stuff done in the organization through relationships initially. But then you’ll grow to a point and the engineering or whatever org you’re dealing with a grow to a point where you need to basically rely on processes, you can’t rely on just individuals anymore. Individuals come and go from companies, you can write that wrapper code, it’s a one time investment, and it’s done. Even if you have to write it for like 50 different languages. Hopefully, it’s done. And then yes, there’s a maintenance cost. But the cost of trying to maintain all of those relationships and continue those over time, I’d go with wrapper libraries.

Astha Singhal 40:41
I actually think like so in that case, the problem though, then becomes like you wrote something, and then how are you getting adoption. Like it’s maintainance is one problem. But the other problem also is, how are your developers becoming aware of this thing that exists for them to do this thing securely? Right, so like that piece.

So for example, we obviously are very automation heavy in our overall program, but then the partnership time that now in a more mature program that we spend with the teams, it’s really around sort of like adoption of those security building blocks that we have built for them, and then try to sort of inform our guidance as much as possible with the data that we have about their applications. But you do need sort of like the relationship with the teams to be able to get them to do the right thing.

Devdatta Akhawe 41:28
I think the other issue I have with processes is that I think processes makes sense when you have a certain maturity around the tech and your platform. In the if the process is like, think about XSS. That’s not a great like process, right? So I think processes makes it much more effective once the core parts of like your tech stack, your platform security, it like if you don’t have a secret management system, a process that says, “What are you going to do about secrets management?” is not great, I would rather focus first on writing those secrets management systems. And then once you have those building blocks, and the issue is people aren’t using them? In that case, yes. Like the processes can help. But like I do think, there’s a lot of engineering that has to go first for the processes to be effective.

Divya Dwarakanath 42:10
So I think, going back to the question, whether I would use a library or like a scanning tool, or whatever, really depends on the problem you’re trying to solve, what the best way is to solve them what sorts of resources you have, and the current maturity of what you’re trying to do.

Like to give you an example. For the longest time, we had one web app at snap, everything else was mobile. And so when we heard that there was going to be more efforts to do to build more web apps, that was it gave us an opportunity to go and be like, okay, this is a good opportunity for us to convince the web dev group with that we need a framework that can take care of XSS, and just sort of agree that the framework level protection is best for us, because we just starting off right now. So it really depends on your situation and what you have at hand.

Clint Gibler 43:12
Okay, great. Yeah, let’s go back to the audience. Yes, sir.

John Heasman 43:16 You, you try and align their personal interests with the interests of the team. So one of the things we tried it out last year was getting some of the more junior people on the team to submit to conferences. And, you know, we’re DocuSign, we don’t create autonomous vehicles or, you know, flying cars or anything. So I wanted to align their research with actual problems we’re solving. So that was the key sort of, I wanted to have them have their research time and feel really excited about that and get to present at a conference, but I wanted that work to be useful for what we were doing. So that’s one way just sort of listening to people starting out on their journey in security. And maybe they don’t know where they want to end up, you can suggest some steps to them, and just aligning those steps with what you’re trying to do with the company and your team.

Divya Dwarakanath 44:07
Yeah, I would say put people in the right roles, give them the autonomy to think big and to tackle big security problems. If people think they’re working on cool stuff, they won’t leave.

Astha Singhal 44:22
Sorry, go ahead.

Doug DePerry 44:24
Yeah, I mean, I would echo I would agree with that. I mean, it’s not as fun. But yeah, I would agree with that. And also, I mean, communication helps a lot. And sometimes you’ve just gotta be prepared for it. Right, like, tech moves pretty quick. People jump around, especially in security, you know, one year, two years, like it’s common, you know, be prepared for it. Yeah, you got to try to keep people happy. But sometimes you have you have a job to do as well. And if you can kind of balance those things with research or just, you know, like, knowing what the person likes, what they don’t what they dislike. But then sometimes it just sometimes they just want to move on. Maybe they completely disagree with the with the philosophy of your team, and there’s nothing you’re going to do to change that. Right.

Astha Singhal 45:02
Yeah, so I definitely agree that there is something called healthy attrition. And I think that’s fine. I think the biggest important thing is you need to build a culture that people want to work in. Because that’s like, really, the biggest reasons I think people leave places is one, if they’re not getting to work on things they want to work on, they’re not getting to grow, or if they’re just not identifying with the culture of the company. So I think that is really the most important thing.

Devdatta Akhawe 45:27
I think I see it as like, there’s nothing unique about security, like, I think talking to your HR partners, they like culture, and how to have a supportive environment, how to have how to challenge people, these are all problems all teams face. And I think I don’t see a lot that is unique to security, I think we should talk to people who have been doing this for a while or experts and get their help. I certainly do. I think the only thing maybe unique is like conferences like these. And so I’m really worried someone from my team is going to leave after looking at the view here and says, hopefully not.

Divya Dwarakanath 45:05
We’re hiring.

Devdatta Akhawe 45:07
There’s no company called Snap.

John Heasman 46:09
so I would also argue, maybe you want people to leave the team, like I’ve had an AppSec engineer, go into engineering at DocuSign. And that’s great. You know, alumni, you want alumni of app sec, throughout the company so that you know, you’re in safe hands, then that person will always come to us. He knows our processes and procedures. We don’t we don’t need to worry about anything that he’s working on.

I saw a presentation a couple of years ago, I actually can’t remember who gave it but the idea is stuck with me, they showed sort of the whole info sec org, like a subway map, all the different places you can come in, different teams, both internally, and then sort of roles you can come into externally. And then the different change points where you can maybe switch from one team and InfoSec to another like risk management to InfoSec or something like that. And then all the ways you might leave InfoSec and go like either into the big wide world or into the company. So I thought that was a really interesting idea.

You know, we want to be, as I said earlier, I built my AppSec team by taking people from QA, tech ops, at DocuSign, I think of some of the partner teams in info sec. And we’ve taken people from legal, we’ve taken people from customer service. So I think you got to think of it like that, rather than just how do we hold on to these people at all cost?

Clint Gibler 47:21
Okay, I think we have time for one more quick question. Yeah.

How do you automate inventory and discovery?

Astha Singhal 47:30
So yeah, in this conference, if you’ve talked to anybody from Netflix, I’m sure that one of the people from the security team have mentioned asset inventory. So but yeah, no, I think like that is definitely we’re realizing more and more that that is a one of the most foundational components of kind of running a successful security program. So we’re investing pretty heavily into that.

For us, we are relying a lot of the cloud infrastructure tooling that exists within Netflix that the company uses for releasing software. And that is definitely a luxury to be able to rely on that. And then the problems we’re trying to work on now is like, how do we sort of gather all that data from the different data sources, correlate, and then come up with sort of the intelligence on you know, like, the risk of the app? Or why should we care about something. So the value that we as security people can add from the standpoint of putting a security lens on top of all of that data?

One Sentence Takeaways

Clint Gibler 48:26
I think we’re actually out of time, but let’s just maybe leave every if anyone wants to give sort of like a one sentence takeaway from each of you.

Doug DePerry 48:36
Ddatadog provides unmatched asset management capabilities, whether you’re in the cloud or in the data-

Clint Gibler 48:43
Let me clarify: some security automation advice, not a product pitch.

Doug DePerry 48:50
We’re hiring as well.

So I think my advice here is is to gain visibility, and then automate, you have to know what you’re working with. We kind of did things a little bit opposite, and I don’t, and I think it’s worked out pretty well for us, because we had some really talented security engineers that can that can write some really good code. And so that’s worked out for us. And we’re kind of gaining more of the visibility stuff, now. But my advice be get the visibility first, and then start automating once you know exactly the data sets that you’re working with.

Clint Gibler 49:24
Great, awesome.

Divya Dwarakanath 49:27
He read my mind.

Yeah, I would say like, prioritize your biggest risks and just automate yourself out of each and every one of them and your automation, your solution, and when and where you implement that is going to be very different and do whatever it takes for your org.

John Heasman 49:54
So one thing, one topic we didn’t really talk about, which makes me really sad, is sort of that telemetry aspect, logging and monitoring. So that’s hugely important.

You know that these type of discussions tend to focus on assuming best efforts to find all bugs when we ship code and not sort of around the acceptance, of course, there’s going to be bugs, people are going to find those bugs, and you really need to know what’s going on in your inside your application.

So at DocuSign, anytime an engineer makes a fix, anytime they go into the code to change something, we really advocate that they add telemetry that will help the security team. And then a really interesting discussion is then how to train your Incident Response Team that are used to taking events from a SIEM, for example, and making them understand the alerts that are coming out of applications. I don’t necessarily mean just things like plugging in a WAF and getting the alerts against you. Millions of people doing SQL injection, I mean, things like actual alerts generated from within your code, things like, “Hey, we saw this SAML token coming in, and it had multiple assertions and it”, just sort of weird things and how you respond to those.

Astha Singhal 50:59
Also see, you can like hitch your security wagon to developer productivity, because a lot of the productivity engineering teams are building tooling that has like a big downstream impact. So if you can get your security tooling and things like that, like in by default in those tools, or if you can rely on some of the visibility things that are building, I think that’s like usually a great way to be able to give yourself a head start because developers want to be more productive. So they are really committed to that tooling. And did you know that there are security people on the Netflix team in LA now?

Devdatta Akhawe 51:32
Won’t they all leave like the weather in North California to come here? I think the quote from one of my favorite books, there’s no silver bullet, there’s just lots and lots of lead bullets. And so just don’t overthink it. Just do anything that like what seems like a good idea, do it move on to more things, do more things stop thinking about looking for a perfect silver bullet, and an added a abroad program and a series of steps will provide good security.

Clint Gibler 52:02
Awesome. Well, thank you all so much for your time. We’ll be around the chat more after.

Stay in Touch!

If you have any feedback, questions, or comments about this post, please reach out! We’d love to chat.