Startup Security: Starting a Security Program at a Startup

Evan Johnson, Senior Security Engineer, Cloudflare
abstract slides video

In this talk, Evan describes what it’s like being the first security hire at a startup, how to be successful (relationships, security culture, compromise and continuous improvement), what should inform your priorities, where to focus to make an immediate impact, and time sinks to avoid.

This talk won’t help you build Google Project Zero in the next year, but it will give you a set of guiding principles you can reference.

If you’re lucky enough to be the first person working on security at a company then you have a great privilege and great responsibility. Successful or not, it’s up to you.

You will set the security tone of the company, and interactions with you will shape how your colleagues view the security team, potentially for years to come.

Evan’s Background

Evan has only worked at startups - first at LastPass, then as the first security engineer at Segment, where he worked for a bit over a year, then he was the first security engineer at Cloudflare, which he joined in 2015.

What is a Security Team?

Depending on the company and who you ask, a vast range of responsibilities fall under the security team’s purview. Here are some examples Evan came across.

In a large company, each of these functions may have a separate team, but in Evan’s experience, at startups, you might have 5 people for 12 different functions.

Security at a SaaS business

Evan breaks down “doing security” in a SaaS business as “protecting, monitoring, responding, and governing” the following categories:

• Production: SaaS companies generally have an “engineering first” culture. Leaders generally are on the product and engineering side, which gets much of the attention.
  • This area includes AppSec, infrastructure security, cloud security, and whatever type of engineering function you need to secure production.
• Corporate: How people access systems, how people do their jobs, endpoint security (e.g. employee laptops), providing employees a safe and secure place to do their job, onboarding and offboarding, etc.
• Business Operations
• Unexpected: Security people get pulled in to a lot of weird things in smaller companies, ranging from production vulnerabilities and incidents to HR issues that have security-relevance.

It’s also critical for the security team to communicate what they’re doing to their colleagues, other departments, the board, and the company’s leadership. Security progress can be hard to measure internally, but it’s an important aspect of your job, as is being accurate and setting the right expectations externally.

Security at Startups

Joining a startup is a great way early in your career to get more responsibility than anybody should probably give you.

If you want to be successful at a startup, there are 3 core aspects:

1. Relationships
2. Security Culture
3. Compromise and Continuous Improvement

1. Relationships

When you are building a security program from scratch
You have customers, just like your company- your coworkers! Building strong relationships with your engineers is hugely important and it’s the secret to success. Without this, you’ll have a hard time getting engineers to work with you.

Additionally, building strong relationships with everyone else in the company is important. Try baking security in to the on-boarding and off-boarding flow, as it is a great place to add security controls and meet your colleagues.

Your relationships are defined in times of crisis.

When issues arise, assure people it’s okay, you’ll fix things together.

Handling hard situations

• As much as possible, be level-headed, calm, and methodical in times of crisis.
• Evan was a security engineer at Cloudflare during Cloudbleed, a bug so important it has its own Wikipedia page. The sales team was really worried about conversations with customers, but the security team joined these calls, were very clear and open about the issue and what they had done to resolve it, and this led many of the conversations to go quite well.

2. Security Culture

Think about the type of security culture you want to build within your company. How far is too far?

If you pull pranks on unlocked laptops, does that foster the type of trust between your team and the rest of the company that you want?

3 tips to building security culture:

• Smile, even if you’re in introvert. It sounds simple, but it makes a difference.
• Be someone people like to be around - don’t only come to dev teams with asks and bad news.
• You should be making technical changes, that is, have an engineering-first culture on security teams.
  • Building things earns trust with engineers who are building your product, as it shows you can contribute too.

3. Compromise and Continuous Improvement

Meet people where they are and try to continuously work with them to make them more secure. If they are really insecure to begin with, then it’s your responsibility to help them improve.

It’s no one’s fault that they have bad practices or are insecure, but it’s your responsibility to do something about it.

Realize that it can sometimes take years to get the kind of traction you want to fix underlying issues. That’s OK.

Security is one of the only parts of the company that has a 30,000 ft view of everything.

Remember: Security is usually a (big) cost center for your company and it can slow down development. There are people who think security teams are evil and don’t like to work with them.

Your First Day

You enter the startup’s office. There are huge windows with natural light and edgy but not too edgy art on the walls. Exposed brick and pipes are plentiful, it’s a bit hard to think over the cacophony of the open office floor plan, and there’s more avocado toast then you can shake a term sheet at.

The co-founder sits you down and asks:

So, what challenge are you going to tackle first?

How Security Starts

It’s not uncommon for companies to have an incident or two and/or get to a certain size where they realize they need someone to work on security full time. You might want to ask during the interview, or at least when you show up on the first day, what prompted investing in security now?

Remember:

• You were hired because you are the expert: people will listen to you.
• You can do whatever you would like: whether it’s good for security or not.
• You’ll have little internal guidance.

What Should Inform Your Priorities

All of the following influence what you prioritize first:

B2B vs B2C
Who are your customers and what do they want from your company / the security team? To Evan, this is the biggest thing that will inform your priorities.

If you’re a B2B SaaS business, you’ll need compliance certifications in order for your sales team to continue selling the product to bigger and bigger customers. Sometimes a company can start as B2C and become more B2B focused, such as Dropbox.

Company Size
If you’re the first security person joining a 500 person team vs a 100 person team, you’ll likely prioritize different things. If the team is already large, you may want to focus on hiring other members on your team to scale your efforts.

Customer Base
Who your customers are within B2B or B2C also influences things, for example, selling HR software to banks vs. marketing software for solo consultants.

Product
There are different expectations of the security bar of your company based on your product. If you are selling a security product, try not to make it an insecurity product like most AV.

Engineering Velocity
If your company has a move fast and break things mentality then the security team needs a different approach than if the culture is a low and slow type of company.

Company culture
Some cultures are really open to security people joining, other don’t understand why security is important. Every company is different.

In summary, companies care more or less about different things and will have different areas of risk. For example: a social network with lots of PII vs a bank with financial information vs a mobile gaming app.

Startup Security Playbook

Evan breaks where you should focus your efforts into 4 core areas:

The common denominator of all of these is that they’re short in scope. You can get 95% of the way to at least initially addressing all of these in a quarter.

Security Engineering

This includes product / application security, infrastructure security, and cloud security.

SDLC and Security Design Reviews with engineers
This is a really big and important area. Starting to work with engineers and embedding yourself in how they work pays major dividends later.

If there isn’t a current SDLC structure, you can do inbound only. You can spread word that you do code reviews and help with threat modeling and recommend it to developers. Show value to them and word will spread. Your ad-hoc process might not have full coverage, but it’s good to start and will still be valuable.

Understanding your tech stack by engineering
Evan feels strongly that security is becoming more like engineering. You need engineers to build the things you want for your security team.

If you want to make a difference at a startup with the way people are building software, you need to build software.

If you want to learn about how your tech stack works at a deep level, you need to build software.

As mentioned previously, a great way to build relationships with engineers is to work with them and have them see you build things as well.

How your manage secrets, keys, and customer secrets
Take inventory of all of your really critical assets.

• Secrets - Do you have a management system for secrets? How well does it work? Are people happy with it? Do you need to roll out Vault or some other secrets management system?
• API Keys - API keys are the keys to your 3rd party SaaS providers. How are they used in prod? How are they shared between developers? What API keys do devs have access to? Do you have engineers with prod AWS/GCP/Azure API keys on their laptops?

You can have a big impact in a short amount of time here.

Bug Bounty
Evan has never had a truly great experience with bug bounty, though it can be a great tool in your toolbelt. Don’t rush into having a bug bounty, wait until you have the bandwidth to quickly and efficiently address submissions and resolve reported issues.

You need to make sure that you have a goal of what types of issues you want out of your bug bounty, and your bug bounty is set up to get that output. Otherwise, you will just waste cycles.

Detection & Response / Incident Response

Detection and Response is one of the hardest areas to get traction, potentially it can seem like you aren’t making much headway until your capabilities are really good.

It’s also something that spans a ton of different domains: production, corporate, incidents, applications, infrastructures, SaaS… it’s everything.

Basic Incident Response Plan
How you handle incidents is really important. Have a plan, get people to understand that plan, and make sure you are looped in when things go awry.

Set up a commmunication channel- people will start reporting things immediately, especially if you don’t already have some logging and monitoring set up. Create a way for people to tell you when things are on fire.

What are your top security signals for the organization?
What really matters for security, and how do you get insights in to them? You are not going to have 100% coverage over D&R for a long time.

Consider starting here:

• Monitor AWS key usage
• Monitor access to applications in your identity provider
• Monitor DNS at your corporate offices (will find things like malware C2, malicious browser extensions, etc.)

As you built out the program, you’ll find more issues, more things for you to do. Have a lightweight approach that you can extend over time.

Establish a communication channel with rest of company
How do people talk with you? How do people get ahold of you when they need you? This can be as simple as an email alias.

Logging Strategy
Over the long term you’re going to need to think about where all of your logs are going. Are you using Splunk, Elasticsearch, rolling your own?

Compliance

Evan has the least amount of experience in compliance.

Public facing security docs are great
Something you can put on your website with technical details about the security things you’ve done that people can reference. Have a security page and a security@ alias for people to report bugs.

Knowledge Base
The best use of your time is not completing questionaires for sales teams. Find a way to make it self-service, so people can complete the questionnaires themselves. If you spend all your time on questionaires then it will be hard to make real strides forward.

Understand existing commitments
Before security people join a startup it’s common for the business to make commitments to future compliance standards that they might not be ready for, but might not have any idea how hard it will be. Sometimes that’s why you were hired in the first place.

“We will have SOC2 Type 2 in 3 months.” That’s not a good situation to be in. You need to understand what commitments your company has made when you join. Ask. It’s not uncommon to have guarantees like this in enterprise contracts.

GDPR and current laws
Make sure you comply with all of the relevant laws.

Corporate Security

Identity and Access Management
This isn’t just a corporate security problem, it’s also a production problem. Production takes years to fix but corp is something you can do in a few quarters. You need a way to manage your applications and access to them.

Endpoint Security
Tablestakes for the long term. It’s better to get this done sooner rather than later, because it’s easier the smaller your company is.

On-boarding and Off-boarding
You can bake a lot of security (and hopefully usability) into these. They’re also tightly coupled with Identity and Access Management: do you remove people from prod? Do you give people the apps they need to do their job?

When on-boarding is a bad experience, studies show people are much less likely to be successful in their role, so it pays off to spend time on making on-boarding smooth.

Workplace security
How do you protect people in your space? How do people badge in? Do they have to wear badges? Do you have doors propped open? Many startups start in small lofts and when they get a bigger space, they’re not used to hanndling these types of issues. Have procedures and policies for the ways visitors visit.

Personal Stories

segmentio/aws-okta - success

At Segment, they deleted every engineer’s AWS key that they used on a day-to-day basis and gave them an equivalent role that they could access through Okta. This aws-okta tool is a drop-in replacement for aws-vault, which engineers were previously using.

Why was this such a success:

• It raised Segment’s security posture massively.
• The dev UX was really fast and smooth.
• It was a massive change that engineering went to 100% adoption in 2-3 weeks of the security team working on it.

segmentio/chamber - failure

As the security team, it’s easy to say that you’ll handle all of the secrets for devs. But, Evan quickly found that people assumed he’d also handle rotation, ownership, managing the secrets, etc.

This project dragged on until he cut the scope and reset the expectations of engineering.

Key Takeaways

The two most important takeaways are:

• Focus on things you can complete.
• Focus on the biggest problems.

Otherwise you’ll never finish anything. Knock out the big things that you can finish.

Avoid taking on things that require lots of upkeep when you are a small team

• Bug bounty takes a lot of upkeep. Buy the triaging from the program you’re using.
• Avoid overcommitting to design reviews, arch reviews, etc. If you spend all your time doing those you will never be able to focus on making security better over the long run.

How are you more secure this month compared to last month?
Everyone should be asking this every month. What is built now that wasn’t there a month ago?

Startups are not for everyone
If you don’t have executive support, you should find a new company.

Security company? You better dogfood
If it’s good enough for your customers, it should be good enough for you.

Remember to keep it simple. Don’t focus on fancy solutions. Focus on the basics.

Focus on things that actually have a meaningful and measurable impact, instead of just fun projects you want to work on.

Questions

If you’re a developer, how can you start getting involved in security?

As you go about your job, see if there are small security tasks you can take on to improve your team or company’s security. Talk with your manager and/or the security team about it, they’ll likely be happy to have you help out.

Do you have any strategies for helping future-proof your company’s security as it grows rapidly?

This is really hard and there’s no magic solution. Being able to brutally prioritize is key: determining which fires you can let continue to burn while you work on more pressing matters.

When you come into a company that already has all of this infrastructure in production and assets, how do you take inventory of what you already have (code) as well as your Internet-facing attack surface?

Evan’s generally accomplished this by working really closely with people who know these things; there are always a few people who know where all the bodies are buried, and they’ll tell you. They’ll be able to tell you valuable things like, “I did X for this security reason,” which will help you figure out where things are at and pick it up from there.

It’s essential to build trust with these people so you can keep asking them these questions. Learn who to ask.

Do you have any advice on startups with distributed teams?

Cloudflare has a strong in office culture, but they do have other offices, for example, in London and Singapore. Evan tries to address this by regularly visiting other offices, getting face time in in person, and waking up early / working late to address time zone issues.