Game On! Adding Privacy to Threat Modeling

Adam Shostack, President, Shostack & Associates twitter, linkedin
Mark Vinkovits, Manager, AppSec, LogMeIn twitter, linkedin
abstract slides video

Elevation of Privilege: Background

Adam originally created Elevation of Privilege at Microsoft as a fun and low barrier to entry way to teach threat modeling to junior security engineers and developers. While he’s proud of his book on threat modeling, it’s 600 pages, so not exactly something you hand someone who is just learning.

Adam was inspired by Laurie William’s Protection Poker game, which is an example of increasingly popular “serious game” movement (Wikipedia), which are games designed for a primary purpose other than entertainment, used in industries like defense, education, scientific exploration, health care, emergency management, city planning, engineering, and politics.

Elevation of Privilege: Why a Game?

Games are powerful because:

• They encourage flow: they can be challenging but not too hard that they cause anxiety.

• They require participation: at some point it’s going to be your turn and you have to play, so you need to pay attention, but it passes focus in a way that isn’t aggressive or demanding.

• They provide social permission for playful exploration and disagreement.

  • Games give us permission to act differently than we normally would in a meeting. Normally a junior security engineer might feel uncomfortable asking a senior developer if a given component might be insecure- “Well, this is just a game, so let’s hypothetically think about what it would mean if this was the case.”

  • This similarly makes disagreements less confrontational; instead they can be collaborative and playful.

• They can be fun, which makes it much easier to get developers, QA, and others involved and invested.

Further, Elevation of Privilege produces real threat models, it’s not just a training game you play once and you’re done.

LogMeIn has adopted Elevation of Privilege for all threat modelling sessions for the past 2 years and pretty much every time it’s used it calls out at least 2 - 3 high impact issues that were not previously discovered or called out beforehand. Mark gives an example of an issue it successfully drew out in a piece of a code written by several experienced, principal-level developers.

Elevation of Privilege: How to Play

1. Draw a picture of the new feature or system being discussed.

2. Deal out all the cards.

3. Play hands (once around the table), connecting the threats on a card to the diagram. Cards should stay in the current suit.

4. Play through the deck or until you run out of time.

5. The person appointed to take notes should document all potential issues discussed and create the relevant Jira tickets, discuss items with product managers to determine business relevance and impact, or whatever makes sense for the company and team.

Elevation of Privilege is inspired by the Spades, in that you’re supposed to follow suit, higher numbers win, and trumps when over other cards.

Aces are for threats not listed on the cards, that is, you’ve invented a threat not in the deck.

You get 1 point for each threat and 1 point for winning the hand.

The card suits are from STRIDE, which is a mnemonic for Spoofing, Tampering, Repudiation, Info Disclosure, Denial of Service, and Elevation of Privilege.

An important thing to note is that the cards have been carefully written, through extensive user testing, to hint at what can go wrong in a way that’s accessible and intuitive to non-security professionals.

The Privacy Extension

Extending Elevation of Privilege to include privacy followed the same process building any usable system should follow:

1. Understand your users, the people you’re building for

2. Understand their use cases

3. Rapid prototype a solution - before putting months into it, sketch it out on piece of paper

4. Evaluation your solution based on user feedback and iterate.

Mark emphasized how critical it is to involve the end users as early as possible in the process. Otherwise, you’ll waste significant time building things they won’t want to use. This process is called user-centered design, user experience (UX), iterative design, etc.

Mark took a number of the Jira issues resulting from some GDPR work they had already done at LogMeIn, abstracted the ideas, and the compiled a list of privacy-related topics that should be discussed during sprint planning.

He ended up building the privacy discussions into Elevation of Privilege itself by adding an additional “Privacy” suit. Here are a few examples:

OH HAI! Another Privacy Extension

After they’d already submitted this talk, they learned that several people from F-Secure had been working on anothe privacy extension of Elevation of Privelege, which is also open source on GitHub.

Their version extends the STRIDE model with TRIM:

• Transport of personal data across geopolitical or contractual boundaries

• Retention and Removal of personal data

• Inference of personal data from other personal data, for example, through correlation

• Minimisation of personal data and its use

References

You can get the Elevation of Privilege card deck for free (print it yourself) or have Agile Stationery put it on some nice sturdy cards for you. You can also see a transcript of this talk on Agile Stationery here.

Both the original and the privacy extension are freely available and licensed under Creative Commons. Get the original here, Mark’s privacy version here, and the F-Secure privacy version here.

Questions

Adam generally asked people, “Give me 15 minutes of your time where you suspend your sense of disbelief. We’ll play a bit, and then after 15 minutes, we’ll decide if it’s not working.” He gives many trainings, and has always found it to work.

Adam generally tries to read the room: do I have people who need some coaching and help, or do I have people who will be engaged by the competition and try to beat the game’s creator? He’ll then adjust his style to be collaborative or cutthroat as appropriate. He prefers to be collaborative, get people engaged, and he’ll just act as a coach.

Mark has found that when he doesn’t deal cards to himself, it conveys to the developers that this isn’t just going to be some AppSec person telling them what to do, it’s going to be a process where they take ownership, be self-directed, and find out for themselves what they need to do.