Startup Security: Starting a Security Program at a Startup

Evan Johnson, Senior Security Engineer, Cloudflare twitter, linkedin
summary abstract slides video

In this talk, Evan describes what it’s like being the first security hire at a startup, how to be successful (relationships, security culture, compromise and continuous improvement), what should inform your priorities, where to focus to make an immediate impact, and time sinks to avoid.

This talk won’t help you build Google Project Zero in the next year, but it will give you a set of guiding principles you can reference.

You will set the security tone of the company, and interactions with you will shape how your colleagues view the security team, potentially for years to come.

Evan’s Background

Evan has only worked at startups - first at LastPass, then as the first security engineer at Segment, then he was the first security engineer at Cloudflare, which he joined in 2015.

Security at a SaaS business

Evan breaks down “doing security” in a SaaS business as “protecting, monitoring, responding, and governing” the following categories: production, corporate, business operations, and the unexpected.

It’s also critical for the security team to communicate what they’re doing to their colleagues, other departments, the board, and the company’s leadership. Security progress can be hard to measure internally, but it’s an important aspect of your job, as is being accurate and setting the right expectations externally.

Security at Startups

If you want to be successful at a startup, there are 3 core aspects:

1. Relationships

2. Security Culture

3. Compromise and Continuous Improvement

1. Relationships

When you are building a security program from scratch
Your coworkers are your customers! Building strong relationships with your engineers is the key to success. Without this, you’ll have a hard time getting engineers to work with you.

Building strong relationships with everyone else in the company is important as well. Try baking security in to the on-boarding and off-boarding flow, as it’s a great place to add security controls and meet your colleagues.

When issues arise, assure people it’s okay, you’ll fix things together. As much as possible, be level-headed, calm, and methodical in times of crisis.

2. Security Culture

Think about the type of security culture you want to build within your company. If you pull pranks on unlocked laptops, does that foster the type of trust between your team and the rest of the company that you want?

3 tips to building security culture:

• Smile, even if you’re in introvert.

• Be someone people like to be around.

• You should be making technical changes (i.e. have an engineering-first culture on security teams).

  • Building things earns trust with engineers who are building your product, as it shows you can contribute too.

3. Compromise and Continuous Improvement

Meet people where they are and try to continuously work with them to make them more secure. It’s no one’s fault that they have bad practices or are insecure, but it’s your responsibility to do something about it.

Realize that it can sometimes take years to get the kind of traction you want to fix underlying issues. That’s OK.

Your First Day

You enter the startup’s office. There are huge windows with natural light and edgy but not too edgy art on the walls. Exposed brick and pipes are plentiful, it’s a bit hard to think over the cacophony of the open office floor plan, and there’s more avocado toast then you can shake a term sheet at.

The co-founder sits you down and asks:

How Security Starts

It’s not uncommon for companies to have an incident or two and/or get to a certain size where they realize they need someone to work on security full time. You might want to ask during the interview, or at least when you show up on the first day, what prompted investing in security now?

Remember:

• You were hired because you are the expert: people will listen to you.

• You can do whatever you would like: whether it’s good for security or not.

• You’ll have little internal guidance.

What Should Inform Your Priorities

All of the following influence what you prioritize first:

B2B vs B2C
This is the biggest thing that will inform your priorities. Who are your customers and what do they want from your company / the security team?

If you’re a B2B SaaS business, you’ll need compliance certifications in order for your sales team to continue selling the product to bigger and bigger customers.

Company Size
If you’re the first security person joining a 500 person team vs a 100 person team, you’ll likely prioritize different things. If the team is already large, you may want to focus on hiring other members on your team to scale your efforts.

Customer Base
Who your customers are within B2B or B2C also influences things, for example, selling HR software to banks vs. marketing software for solo consultants.

Product
There are different expectations of the security bar of your company based on what your product is.

Engineering Velocity
If your company has a move fast and break things mentality then the security team needs a different approach than if the culture is a low and slow type of company.

Company culture
Some cultures are really open to security people joining, other don’t understand why security is important. Every company is different.

In summary, companies care more or less about different things and will have different areas of risk.

Startup Security Playbook

Evan breaks where you should focus your efforts into 4 core areas:

The common denominator of all of these is that they’re short in scope. You can get 95% of the way to at least initially addressing all of these in a quarter.

Security Engineering

This includes product / application security, infrastructure security, and cloud security.

SDLC and Security Design Reviews with engineers
Starting to work with engineers and embedding yourself in how they work pays major dividends later. If there isn’t a current SDLC structure, you can do inbound only. Offer to do code review and threat modeling, show value, and word will spread. Your ad-hoc process won’t have full coverage, but it’s a good start.

Understanding your tech stack by engineering

A great way to build relationships with engineers is to work with them and have them see you build things as well.

How your manage secrets, keys, and customer secrets
Take inventory of all of your really critical assets.

• Secrets - Do you have a management system for secrets? Are people happy with it? Do you need to roll out Vault or some other secrets management system?

• API Keys - How are they used in prod? How are they shared between developers? What API keys do devs have access to? Do you have engineers with prod cloud platform API keys on their laptops?

You can have a big impact in a short amount of time here.

Bug Bounty
Don’t rush into having a bug bounty, wait until you have the bandwidth to quickly and efficiently address submissions and resolve reported issues.

Detection & Response / Incident Response

Detection and Response is one of the hardest areas to get traction. It’s also something that spans a ton of different domains: production, corporate, incidents, applications, infrastructures, SaaS…

Basic Incident Response Plan
Have a plan, get people to understand that plan, and make sure you are looped in when things go awry.

Set up a commmunication channel- people will start reporting things immediately, especially if you don’t already have some logging and monitoring set up. Create a way for people to tell you when things are on fire.

What are your top security signals for the organization?
What really matters for security, and how do you get insights in to them?

Consider starting with monitoring AWS key usage, access to applications in your identity provider, and DNS at corporate offices.

Establish a communication channel with rest of company
How do people talk with you? How do people get ahold of you when they need you? This can be as simple as an email alias.

Logging Strategy
Where are you going to store logs over the long term?

Compliance

Public facing security docs are great
Something you can put on your website with technical details about the security things you’ve done that people can reference. Have a security page and a security@ alias for people to report bugs.

Knowledge Base
The best use of your time is not completing questionaires for sales teams. Find a way to make it self-service.

Understand existing commitments
Before security people join a startup it’s common for the business to make commitments to future compliance standards that they might not be ready for, but might not have any idea how hard it will be. Sometimes that’s why you were hired in the first place.

Ask management what compliance commitments your company has made.

GDPR and current laws
Make sure you comply with all of the relevant laws.

Corporate Security

Identity and Access Management
You need a way to manage your applications and access to them. Corp and prod are both important, but corp may be easier to address first.

Endpoint Security
Tablestakes for the long term. It’s better to get this done sooner rather than later, because it’s easier the smaller your company is.

On-boarding and Off-boarding
You can bake a lot of security (and hopefully usability) into these. They’re also tightly coupled with Identity and Access Management: do you remove people from prod when they no longer need access?

Workplace security
How do you protect people in your space? How do people badge in? Do they have to wear badges? Do you have doors propped open? Many startups start in small lofts and when they get a bigger space, they’re not used to hanndling these types of issues. Have procedures and policies for the ways visitors visit.

Personal Stories

segmentio/aws-okta - success

At Segment, they deleted every engineer’s AWS key that they used on a day-to-day basis and gave them an equivalent role that they could access through Okta. This aws-okta tool is a drop-in replacement for aws-vault, which engineers were previously using.

Why was this such a success:

• It raised Segment’s security posture massively.

• The dev UX was really fast and smooth.

• It was a massive change that engineering went to 100% adoption in 2-3 weeks of the security team working on it.

segmentio/chamber - failure

As the security team, it’s easy to say that you’ll handle all of the secrets for devs. But, Evan quickly found that people assumed he’d also handle rotation, ownership, managing the secrets, etc.

This project dragged on until he cut the scope and reset the expectations of engineering.

Key Takeaways

The two most important takeaways are:

• Focus on things you can complete.

• Focus on the biggest problems.

Otherwise you’ll never finish anything. Knock out the big things that you can finish.

Avoid taking on things that require lots of upkeep when you are a small team

• Bug bounty takes a lot of upkeep. Buy the triaging from the program you’re using.

• Avoid overcommitting to design reviews, arch reviews, etc. If you spend all your time doing those you will never be able to focus on making security better over the long run.

How are you more secure this month compared to last month?
Everyone should be asking this every month. What is built now that wasn’t there a month ago?

Startups are not for everyone
If you don’t have executive support, you should find a new company.

Security company? You better dogfood
If it’s good enough for your customers, it should be good enough for you.

Questions

As you go about your job, see if there are small security tasks you can take on to improve your team or company’s security. Talk with your manager and/or the security team about it, they’ll likely be happy to have you help out.

This is really hard and there’s no magic solution. Being able to brutally prioritize is key: determining which fires you can let continue to burn while you work on more pressing matters.

Evan’s generally accomplished this by working really closely with people who know these things; there are always a few people who know where all the bodies are buried, and they’ll tell you. They’ll be able to tell you valuable things like, “I did X for this security reason,” which will help you figure out where things are at and pick it up from there.

It’s essential to build trust with these people so you can keep asking them these questions. Learn who to ask.

Cloudflare has a strong in office culture, but they do have other offices, for example, in London and Singapore. Evan tries to address this by regularly visiting other offices, getting face time in in person, and waking up early / working late to address time zone issues.