- tl;dr sec
- The Call is Coming From Inside the House: Lessons in Securing Internal Apps
The Call is Coming From Inside the House: Lessons in Securing Internal Apps
A masterclass in the thought process behind and technical details of building scalable defenses; in this case, a proxy to protect heterogenous internal web applications.
Why care about securing internal apps?
In short: they often have access to sensitive data, and it’s technically challenging.
Compared to production, customer-facing applications, internal apps often get neglected by security teams. However, these internal apps often expose sensitive data or functionality, like a debug panel for production systems or a business analytis app with sensitive company data.
Unfortunately we can’t just firewall off these internal apps, as they could get accidentally exposed by a network configuration, they can be targeted by an external attacker via CSRF or SSRF, or there may already be an attacker in your environment (hacker or insider threat).
Internal app security is interesting and challenging due to the scale and heterogenity.
Scale: Most companies have a handful of primary production apps but hundreds of internal apps.Heterogenity: The production apps likely use a well-defined set of core tech stacks for which the security team has built secure by default frameworks and other defenses. This is not scalable to dozens of other languages and frameworks. Further, internal apps may be built by people who don’t spend much of their time coding.
When embarking on this work, the Dropbox security team had the following goals:
The defenses should be scalable and agnostic to the backend’s tech stack.
Scalable development and adoption process - development teams’ adoption needs to be as frictionless and self-service as possible.
I want to emphasize how smart these goals are: any backend-agnostic defense would be a great win, and building a great technical solution isn’t enough, you need adoption. And if the security team needs to be involved when the defense is applied by every dev team or project, that’s going to eat up your time for the next… forever.
tl;dr: Add network isolation and enforce that all internal apps must be accessed through a proxy, which becomes a centralized, scalable place to build defenses into.
This approach allows them to:
Add authentication that they log and review and enforce strong 2FA (SSO + U2F)
Access control - check permissions for all requests, using ideas like entry point regulation
Enforce that users are using modern, up-to-date browsers (not vulnerable to known exploits, strong security posture)
Add monitoring, logging, etc.
Other Applications: Later in the talk, Hongyi goes into detail of how they used this proxy approach to add CSRF protections (add
SameSite cookie flags) and XSS protections (
Content-Security-Policy headers + nonces). He also makes the argument that using WAFs on internal networks can be effective, as internal networks shouldn’t have the level of malicious noise that your external websites will.
They have a number of ideas for other applications of the proxy that they haven’t yet done, including CORS allowlists, preventing clickjacking, invariant enforcement, sec-metatdata, channel-bound cookies, canaries, and more.
Reflections on the Approach
Gives you a centralized, scalable place to build defenses into.
App owners don’t have to build these defenses themselves.
It’s easy to deploy patches and changes - you only have to deploy bugs to one place, and it’s a service the security team has control over.
Provides a number of benefits to other security partners: InfraSec, Detection and Response, etc.
Tradeoffs / Considerations:
A team needs to maintain the proxy and be oncall when issues arise.
This should be a team with experience maintaining high uptime systems. Consider partnering with your InfraSec team or an infrastructure team.
Reliability (and performance issues, to a lesser extent) are critical.
Ideally, if the proxy goes down you fail closed (safe), but if there are critical systems behind the proxy, you’ll be blocking people from doing their job.
e.g. If you have a monitoring system behind your proxy, if the proxy goes down, you won’t be able to troubleshoot.
Make developer testing and deployment easy; otherwise, this makes more work for the security team as you need to be heavily involved in adoption. The less friction there is, the more likely developers will want to use it.
Similarly, reduce mental burden on developers as much as possible, using blessed frameworks with defenses built-in and autogenerate security policies.
Make deployment and rollback safe and fast. You’re going to spend a lot of time troubleshooting policy mistakes, so make it quick and easy to reverse those mistakes. Make testing and debugging trivial.
Prioritize for your highest risks when rolling out the proxy; there will always be too much to do, so you have to brutally prioritize. For example CSP might be too much work in your organization, other mitigations may be enough- the migration work migght not be worth the risk reduction provided.
Expect challenges; find trusted partners to try out ideas. In every company, there are people and teams that are more excited about security and willing to be early adopters of new security tools, services, and processes, even if they are currently half-baked, buggy, and high friction to use.
Start with them and iterate based on their feedback. This reduces the cost of failure, as you won’t be burning bridges when your solution isn’t perfect (which may cost you political capital if you bring it to less willing engineering teams before it’s ready). These early adopter teams can give you quick feedback, which allows you to increase your iteration speed.
On iteratively creating policies
Much of the work the Dropbox team ended up doing was refining policies. They could’ve saved a lot of time by initially adding the ability to enforce a relaxed policy while testing a tighter policy.
NIST CyberSecurity Framework
For tieing all of these ideas together into a coherent strategy, Hongyi recommends the NIST CyberSecurity Framework.
The NIST framework identifies 5 main security functions (column headings). Fill out this table and for each category, think about:Who are the teams you need to involve?What is the tech you have to build?What are the processes to create?
This process helps you figure out where are your company’s gaps are and where to invest going forward. This becomes your roadmap, which is also a great way to communicate your strategy to senior management.
Hongyi has borrowed some ideas from Adam Shostack: he considers these efforts like a security investment portfolio, where you’re investing, where you want to change those investments. This is a simple tool but very flexible, you can adapt it as needed.
When you’re determining your AppSec team’s priorities, aim to scale your defenses and your processes.
Internal security engineering can be a great place to experiment with new features and to train up new security engineers. For example, if you’re not sure about CSP or
SameSite cookies, you can deploy them internally first and learn from the experience. It’s also a good way to get experience building safe frameworks, as thet reliability requirements are much lower than in externally facing production contexts.