Securing Third Party Applications at Scale

Ryan Flood, Manager of ProdSec, Salesforce twitter, linkedin
Prashanth Kannan, Product Security Engineer, Salesforce twitter, linkedin 
abstract slides video

Background

The Salesforce AppExchange was launched in 2005 as a way for developers to create new functionality that extends and customizes Salesforce and sell these enhancements to Salesforce customers, like the Google Play store. Apps are written in their custom language, Apex.

There are a number of core challenges in third-party applications, including:

• It’s someone else’s code that you don’t have access to

• You can only test the end product, not evaluate the process (do they have a robust SDLC?)

• Apps will change after you’ve reviewed it and add new features

• They may integrate with other systems that you have no visibility into

1. Define the problem

• Who are the third parties this process is supposed to help (vendors, partners, app developers, customers)?

• What data and other access do they need?

  • Where possible implement restrictions in technical controls rather than policy controls.

Salesforce has built technical controls into the languages app developers use (Apex, Visual Force, Lightning, Lightning web components) and the Salesforce API, keeping apps from breaking out of the sandbox they’re placed in.

Salesforce has an install warning for apps they haven’t reviewed so that customers don’t think the apps have been vetted and place an undue amount of trust in them.

2: Establish baselines

Ensure that the third parties at least considered security: they have some security policy documents, a vulnerability disclosure policy, a security@ email address. Do they have a bug bounty or certs (e.g. ISO 27000/1, SOC 2, PCI)?

Once you decide the “bar to entry” for your marketplace, enforce it across all new apps/partners.

Put your requirement checklist out very publicly; talk about what you expect and why. You can sell this as a way to make the process more scalable and fast for business customers. This will saves you significant communication time.

Salesforce has a gamified learning platform called Trailhead, which can be used to learn about the product. The security team created security-relevant content to put there as well as in the overall Salesforce docs. The security team brings the info they want customers to know wherever they are already.

3. Assess

Now measure these partners against your security bar:

• Security assessments: Ask if they can share any reports from pen tests they’ve received.

• Security tools: Share the open source and commercial tools your company likes, and encourage the partners to scan themselves.

  • As you build automated scanning for your own use in scanning partner code and API endpoints, consider allowing the partners to leverage your tooling as well in a self-service manner.

• Security Terms of Service or Contract Terms - Give yourself a way to hold partners accountable for fixing identified bugs and establish terms for what happens to developers who act in bad faith.

4. Remediation

How companies respond to issues you identify will reveal a lot about their commitment to security: the questions they ask, how fast they resolve issues, how good their fixes are, what they push back on, etc.

Offer support to help them pass the process, after all, this is making customers more secure.

At Salesforce, they:

• Report findings with sufficient context and evidence of the vulnerability and include links to more information, based on industry best practice like OWASP.

• Create disincentives to prevent “brute forcing” the process, for example, longer wait times.

5. Monitor the Ecosystem

Have a single inventory of your third parties indexed on an important business asset that won’t get lost (e.g. vendor contracts, software licenses, API/OAuth credentials).

Track risk and impact related information, such as:

• Usage metrics for an app and who’s using it (e.g. an app may be only used by a certain industry or type of user)

• Prior security reviews - how have they done over time?

• Any automated security information you can get

Salesforce has used this data to influence their product roadmap. For example, if they see that certain vulnerability classes are common or certain APIs tend to be misused, they’ll add additional security controls, documentation, or change how things work to try to make the ecosystem as a whole more secure.

6. Security Check-ups

Periodically you’ll need to re-evaluate approved applications. This can be done after a certain amount of time, after a certain amount of code or API change, or based on the monetary impact of the partner on your business. Decide on a risk-based approach that makes sense for your company.

Check-ups also allow new security/policy measures to be applied to existing apps.

Note: Security is relative to your current environment; as new attack classes or vulnerabilities appear (e.g. HTTP desync attacks, new attacks on TLS), it’s important to re-evaluate.

The Salesforce AppExchange re-reviews apps based on time and impact and they run automated scans when apps are updated. New Salesforce platform launches can trigger checkups as well.

Good to Great

How to improve your review process over time.

Data Analysis on Review information

Data from past reviews of third-party apps is really helpful, for example:

• Maybe your security team’s workload spikes at a certain time of year. You can prepare the security team’s schedule for it or hire contractors.

• Time to review and types/number of vulnerabilities gives insights like: tech stack X tends to have these vulns while language Y tends to other vuln classes. Use this info to train your security engineers.

Label which apps and API endpoints have already been assessed to prevent duplicated work.

Evangelize and educate

Publish your security guidelines internally and externally, evangelize the process and educate the parties involved. Be very clear and consistent with your policy.

Educate your users on how to secure their applications to pass your process

• Document good and bad examples for topics specific to your business domain

• Rely on industry standards like OWASP whenever possible

Have security office hours where your customers can schedule time where you help them verify their architecture choices, address false positives, etc.

Automate to Scale

In the beginning, much of your process will be manual, which is OK. Gradually build automation over time into a unified system that includes metrics reporting and operational support.

Have app devs run the security tools your company uses on their app before they engage with you. This gives them earlier feedback so they can address issues before they even get to you, saving both of you time.

Create an Operations Team

They can be the liaison between third party app devs and your security engineering team.

• This keeps app devs happy with fast responses from a dedicated team.

• This is cost effective, as apps that are missing items to start the review process are caught before they take expensive security engineer time.

• Faster time to resolution: nailing the basics with an operations team means less back-and-forth with third parties

Tools to Kickstart Your process

• Security review management - VulnReport, Faraday

• Source code analysis - Brakeman, Bandit, PMD, vendors like Checkmarx or HP Fortify

• Dependency Analysis - OWASP Dependency Check, retire.js, vendors like Snyk

• Web App Scanning - OWASP ZAP, Nikto, Nmap, vendors like Portswigger Burp Suite