The Call is Coming From Inside the House: Lessons in Securing Internal Apps

Hongyi Hu, Product Security Lead, Dropbox twitter, linkedin
abstract slides video

Why care about securing internal apps?

In short: they often have access to sensitive data, and it’s technically challenging.

Compared to production, customer-facing applications, internal apps often get neglected by security teams. However, these internal apps often expose sensitive data or functionality, like a debug panel for production systems or a business analytis app with sensitive company data.

Unfortunately we can’t just firewall off these internal apps, as they could get accidentally exposed by a network configuration, they can be targeted by an external attacker via CSRF or SSRF, or there may already be an attacker in your environment (hacker or insider threat).

Internal app security is interesting and challenging due to the scale and heterogenity.

When embarking on this work, the Dropbox security team had the following goals:

• The defenses should be scalable and agnostic to the backend’s tech stack.

• Scalable development and adoption process - development teams’ adoption needs to be as frictionless and self-service as possible.

The Approach

tl;dr: Add network isolation and enforce that all internal apps must be accessed through a proxy, which becomes a centralized, scalable place to build defenses into.

This approach allows them to:

• Add authentication that they log and review and enforce strong 2FA (SSO + U2F)

• Access control - check permissions for all requests, using ideas like entry point regulation

• Enforce that users are using modern, up-to-date browsers (not vulnerable to known exploits, strong security posture)

• Add monitoring, logging, etc.

Other Applications: Later in the talk, Hongyi goes into detail of how they used this proxy approach to add CSRF protections (add SameSite cookie flags) and XSS protections (Content-Security-Policy headers + nonces). He also makes the argument that using WAFs on internal networks can be effective, as internal networks shouldn’t have the level of malicious noise that your external websites will.

They have a number of ideas for other applications of the proxy that they haven’t yet done, including CORS allowlists, preventing clickjacking, invariant enforcement, sec-metatdata, channel-bound cookies, canaries, and more.

Reflections on the Approach

Benefits:

• Gives you a centralized, scalable place to build defenses into.

• App owners don’t have to build these defenses themselves.

• It’s easy to deploy patches and changes - you only have to deploy bugs to one place, and it’s a service the security team has control over.

• Provides a number of benefits to other security partners: InfraSec, Detection and Response, etc.

Tradeoffs / Considerations:

• A team needs to maintain the proxy and be oncall when issues arise.

  • This should be a team with experience maintaining high uptime systems. Consider partnering with your InfraSec team or an infrastructure team.

• Reliability (and performance issues, to a lesser extent) are critical.

  • Ideally, if the proxy goes down you fail closed (safe), but if there are critical systems behind the proxy, you’ll be blocking people from doing their job.

  • e.g. If you have a monitoring system behind your proxy, if the proxy goes down, you won’t be able to troubleshoot.

Lessons Learned

Make developer testing and deployment easy; otherwise, this makes more work for the security team as you need to be heavily involved in adoption. The less friction there is, the more likely developers will want to use it.

Similarly, reduce mental burden on developers as much as possible, using blessed frameworks with defenses built-in and autogenerate security policies.

Make deployment and rollback safe and fast. You’re going to spend a lot of time troubleshooting policy mistakes, so make it quick and easy to reverse those mistakes. Make testing and debugging trivial.

Prioritize for your highest risks when rolling out the proxy; there will always be too much to do, so you have to brutally prioritize. For example CSP might be too much work in your organization, other mitigations may be enough- the migration work migght not be worth the risk reduction provided.

Expect challenges; find trusted partners to try out ideas. In every company, there are people and teams that are more excited about security and willing to be early adopters of new security tools, services, and processes, even if they are currently half-baked, buggy, and high friction to use.

Start with them and iterate based on their feedback. This reduces the cost of failure, as you won’t be burning bridges when your solution isn’t perfect (which may cost you political capital if you bring it to less willing engineering teams before it’s ready). These early adopter teams can give you quick feedback, which allows you to increase your iteration speed.

NIST CyberSecurity Framework

For tieing all of these ideas together into a coherent strategy, Hongyi recommends the NIST CyberSecurity Framework.

This process helps you figure out where are your company’s gaps are and where to invest going forward. This becomes your roadmap, which is also a great way to communicate your strategy to senior management.

Hongyi has borrowed some ideas from Adam Shostack: he considers these efforts like a security investment portfolio, where you’re investing, where you want to change those investments. This is a simple tool but very flexible, you can adapt it as needed.

Final Thoughts

When you’re determining your AppSec team’s priorities, aim to scale your defenses and your processes.

Internal security engineering can be a great place to experiment with new features and to train up new security engineers. For example, if you’re not sure about CSP or SameSite cookies, you can deploy them internally first and learn from the experience. It’s also a good way to get experience building safe frameworks, as thet reliability requirements are much lower than in externally facing production contexts.