• tl;dr sec
  • Posts
  • [tl;dr sec] #22 - Post AppSec Cali, K8s Security Monitoring at Scale

[tl;dr sec] #22 - Post AppSec Cali, K8s Security Monitoring at Scale

DevSecOps talks & tools from AppSec Cali, PoCs to decrypt WhatsApp messages, Kubernetes monitoring and CTF, Python static analysis tools.


AppSec Cali 2020 was a blast! I met some new cool people, caught up with friends, and got asked more than a few times if I was going to write summaries for all of the talks next year 😅 We’ll see, no promises.

People at the conference as well as on Twitter and LinkedIn after were quite kind and positive about my talk. Like the Grinch, my heart grew three sizes. You can check out my slides here and a Twitter overview here.

Sidenote: It’s kind of stressful editing a Google Slides presentation while seeing ~90 people currently looking at it 😆

🗣️ Feedback Requested: High Leverage Security Engineering Projects?

One of my favorite topics is hearing about neat systems and processes people have built that have had huge security ROI. For example, RepoKid for automating IAM least privilege and Dropbox’s proxy to protect internal web apps.

I’m giving talks at BSidesSF and RSAC in a few weeks that are a continuation of my effort to aggregate and distill scalable modern security best practices.

If your company has built some baller systems or processes that have prevented or mitigated classes of vulnerabilities, or otherwise significantly raised your security posture, I’d really appreciate you emailing me a description or linking me to a talk or blog post about it! 🙏

I’ll cite you and your company if it gets included in the talk, and I’m planning to write a blog post that describes all these projects in one place, like a cookbook manual of cool, impactful things you can build.

📜 In this newsletter...

🔗 Links:

  • AppSec Cali 2020: Slides for DevSecOps talks, new OWASP tool (Nettacker), measuring SAST performance, new OWASP video page

  • Cloud / Containers: Tool to enumerate Docker registries, KubeCon CTF, aligning to NIST CSF in AWS, AWS security docs

  • Kubernetes Security monitoring at scale with Sysdig Falco: Case study on security runtime monitoring of K8s at scale, nice asset inventory and SecOps bot ideas

  • Web Security: A bug bounty program for fixes, protecting yourself from GitHub Action supply chain attacks, exploiting email address parsing

  • Static Analysis (Python): Tool that uses type annotations + an SMT solver to find counterexamples, tool to find taint analysis-style bugs

  • Misc: Global fast Internet via satellites, InfoSec skills matrix, VirusTotal Graph updates

  • Bezos’s Phone: No evidence that implicates MBS, PoC code to decrypt WhatsApp files

  • Politics / Privacy: NIST releases v1 of the Privacy Framework, stats from Apple's national security requests

  • How to Influence Engineering Teams: Great Twitter thread for designers, but also applies to security engineers

AppSec Cali 2020

Over the next few weeks and months more slides and videos will be released. Here are some I’ve come across so far.

DevSecOps Talks

Tools & Misc

Introducing OWASP Nettacker by Sam StepanyanHere’s a demo video of it detecting Citrix CVE-2019-19781 and a list of features on its repo on GitHub:

  • Python Multi Thread & Multi Process Network Information Gathering Vulnerability Scanner

  • Service and Device Detection ( SCADA, Restricted Areas, Routers, HTTP Servers, Logins and Authentications, None-Indexed HTTP, Paradox System, Cameras, Firewalls, UTM, WebMails, VPN, RDP, SSH, FTP, TELNET Services, Proxy Servers and Many Devices like Juniper, Cisco, Switches and many more… )

  • Asset Discovery & Network Service Analysis

  • Services Brute Force Testing

  • Services Vulnerability Testing

  • HTTP/HTTPS Crawling, Fuzzing, Information Gathering and more

A site that aims to provide a data-driven, detailed break down of static analysis tools based on cost, license, supported languages and frameworks, performance, and so forth. Described by Chris Horn in his talk Choosing the right static code analyzers based on hard data.

A more streamlined interface to OWASP videos, built by Jonathan Marcil and others. You can also contribute to the source code here.

Cloud / Containers

Enumerating Docker Registries with go-pillage-registries
Tool release by NCC Group’s Josh Makinen that, given a Docker registry, will enumerate all repos and their tags using Docker’s /v2/_catalog and /v2//tags/list APIs. pilreg was built because on a number of engagements, Josh gained read access to private Docker registries, from which he could extract useful info from images, such as secrets compiled into the container during build time, app logic, config files, etc. For each image pilreg finds, it automatically requests their manifest and configuration and outputs the results code 

KubeCon NA 2019 CTF
Materials for “Attacking and Defending Kubernetes Clusters: A Guided Tour Walkthrough Guide” that was presented at KubeCon NA 2019. There are several challenges that alternate between an attack and defense perspective. (Thx Marco Lancini)

Aligning to the NIST Cybersecurity Framework in the AWS Cloud
Whitepaper from AWS on AWS services that enable alignment with the CSF core functions: Identify, Protect, Detect, Respond, and Recover. Dropbox’s Hongyi Hu called out the NIST CSF as being a really useful framework for planning your security engineering investments during his AppSec Cali 2019 talk. See here for more details.

AWS Security Documentation
A link to AWS’s security documentation for various services all on one page.

Nice blog post from Christian Martorella and Nacho Rasche about how Skyscanner used Falco to monitor >2,000 nodes across 30 different clusters. “Falco is a container Native Runtime Security solution focused in Intrusion and abnormality detection which uses the open source linux Kernel tooling built by Sysdig to generate alerts based on a custom rules and macros engine.”

Falco comes with powerful built-in default rules that can detect things such as a shell being run in a container, a server process spawning a child process of an unexpected type, and a sensitive file, like /etc/shadow, being unexpectedly read.

The post has some nice details about how and why they deployed Falco the way they did. Once they tuned down the initial noisy alerts, they started observing interesting behavior about their environment, such as some containers running as root.

To determine the owner of a pod when alerts fire, they use a system they’ve built that collects ownership and resource information about each service, obtained by scanning the metadata available in the project’s repo, their AWS accounts and a few other places, such as data received from their deployment pipeline.

As I mentioned in my talk, having programmatic visibility into your assets is incredibly useful!

As we continue gathering information from AWS, Github and vulnerability sources like Snyk and Clair, we can also know which AWS roles or infrastructure are linked to the same service, which language and libraries the service is using and whether any of the libraries are affected by vulnerabilities that an attacker could be exploiting.

Finally, all this data is stored in Elasticsearch, so using Kibana we built some dashboards that would easily allow to monitor the number of findings in each cluster, filtering by owner or service criticality, rule triggered and so on.

Skyscanner has over 5,000 services and the security team is 22 people- how do you scale?

They have a ChatSecOps bot that scans received data (in this case, Falco findings + info added by their ownership mapping tool) and contacts the resource owner to inform them of the event. They’re working on having the bot ask the owner if they indeed initiated the action, and if the owner says ‘No,’ it’ll trigger their incident response process.

This process of automatically alerting resource owners to confirm if they initiated actions closely mirrors similar work done by Slack, Dropbox, and Pinterest, as I called out in my talk. This is an effective way to reduce ongoing security engineer operational time requirements, and it’s neat to see this idea put into practice more places.

Web Security

“The first bug bounty board for securing open-source code,” launches publicly Feb 1, 2020. I like the idea of bounties for fixing, not just finding bugs, but I’m not sure how effective this will be in practice for a few reasons: a) finding and fixing require different skills, b) fixing well can require significant contextual understanding of a code base, that a non core dev won’t have, and c) will programs be willing to pay an amount that will incentivize well-written fixes (vs. just band-aids)? Still, I think this is exciting and I’m looking forward to seeing where it goes.

Use GitHub actions at your own risk
Any repo maintainer of a popular GitHub action can change what a branch or tag points to in the future, for example, to insert malicious functionality, such as stealing provided API tokens. To prevent this, specify the exact version you mean (action@sha1). (Thx Marco Lancini)

Exploiting email address parsing with AWS SES
There are certain ambiguities in email specification RFCs such that depending on the language and parsing function used, different email addresses may be returned (e.g. ([email protected])[email protected]). This blog post shows how in some (infrequent) situations you can bypass account email validation/verification by providing an email address that a frontend web app interprets to have one domain while a backend service returns a different one (e.g. AWS SES).

Static Analysis (Python)

pschanely / CrossHair
If you have functions with type annotations and you add some checks in the (defunct) PEP 316 syntax, CrossHair will attempt to find counterexamples by repeatedly calling functions with symbolic inputs and using an SMT solver to explore viable execution paths.

Pyre / Pysa from Facebook
There are now some docs on using Pysa for taint analysis (data flow analysis) to find when user input reaches dangerous functions (e.g. SQL injection, command injection, etc.) The docs describe how you can specify the sources and sinks you care about, as well as sanitizers to reduce false positives. There’s also a Pysa tutorial here.


Starlink is a Big Deal
SpaceX being able to cheaply launch low orbiting satellites may allow Starlink (a subsidiary of SpaceX) to provide fast Internet (600 MBPs is referenced) globally, putting 4 billion more people online. Some interesting predictions about the outcomes of this at the end.

InfoSec Skills Matrix
A chart that visually shows what skills you need (e.g. presentation skills, network fundamentals, web app basics) for which security roles (e.g. pen tester, infrastructure specialist). Useful resource for people thinking about getting in to security or switching to a different speciality. (Thx Daniel Miessler)

VirusTotal Graph++
Blog post on updates to the VirusTotal Graph, a “tool which understands the relationship between files, URLs, domains and IP addresses, and an easy interface to pivot and navigate over them.” Documentation for the API can be seen here and a Python client for the API here. (Thx Marco Lancini)

Bezos’s Phone

As mentioned in tl;dr sec 21, there was quite the hullabaloo about accusations of the Saudi crown prince (MBS) hacking Bezos’s phone by sending him a malicious file over WhatsApp. The initial forensic investigation said, “due to end-to-end encryption employed by WhatsApp, it is virtually impossible to decrypt the contents of the downloader…” Predictably, the Internet said, “Hold my beer.”

Dino Dai Zovi released some PoC scripts that can do just that, and Rob Graham wrote a nice blog post walkthrough of how to do it with PoC code as well. Rob also wrote an overview article for non technical users.

Politics / Privacy

  • The Core enables communications within organizations about privacy protection activities and desired goals.

  • Profiles allow organizations to prioritize the outcomes and activities according to privacy values, the business mission, and risks.

  • Implementation tiers help organizations to optimize the resources that are necessary to manage the risk.

Between Jan. 1 and June 30, 2019, Apple said governments made 31,778 requests for devices – around 500 more than in the first half of 2018. Apple provided data including which customers are associated with which devices, as well as purchases, customer service and repair info.

Apple also provided details on national security-related requests made by the US under the Foreign Intelligence Surveillance Act (FISA). During the six-month period, it received between 0 and 499 requests for FISA non-content data such as subscriber and transactional information across between 11,000 and 11,499 users and accounts; and 0 to 499 FISA content requests such as photos, emails, device backups, contacts and calendars for up to 18,499 users and accounts.

How to Influence Engineering Teams

Great thread by Scott Berkun on influencing people as a designer. Try reading it and replacing “designer” with “security engineer,” as in both cases we’re often trying to influence engineering teams:

If you’re a designer and you want more power, there’s really only a few ways to get it. You either need to change your role or your reputation.

Often it’s a PM (Product manager or Project Manager) that has the power you want. If that’s true, switch roles! Seriously, if that’s who really makes the design decisions, then that’s where you belong.

Or accept that your role is primarily about influence - you’re a kind of consultant and not a decision maker - focus on becoming a master influencer and persuader (learnable skills). Learn to charm. Build allies. Be someone who gives trusted advice that people seek out.

The worst choice is to claim to want more power but not be willing to do anything about it. Everyone wants more influence, the people who get it usually earn it in some way.

Good design only ever happens if the people who make the decisions make good design decisions. You can’t make good things happen if good people don’t have power.

In the end, if you work somewhere with a bad distribution of power there are only three choices: a) work to change it b) work somewhere else or for someone else c) accept it. Doing d) nothing is a bad mental health choice.

The big takeaway might be how ignorant designers often seem to be about the fact that everyone wants more influence! It’s not just designers. Everyone wants to get to make decisions. No one wants to feel left out or ignored. But there are only so many decisions to go around.

A thoughtful designer can learn more about how to be influential from befriending engineers and marketers and making them allies, than talking to more designers and going to more design events. You’re already a design expert. Shouldn’t you study the ppl u want to influence?

Designers are experts at human to computer interaction (HCI). But I’m not so sure about human to human interaction (HHI). Or Human to Group/Org interaction (HGI). It’s HHI and HGI that define how much influence and power you have, not your HCI knowledge.

Influence is first and foremost about understanding other people. Pitching ideas is too. But don’t start from “why don’t I have more influence?” A better frame is “Who do the powerful ppl here let influence them? Why? And how did they earn that influence?”

✉️ Wrapping Up

Have questions, comments, or feedback? Just reply directly, I'd love to hear from you.

If you find this newsletter useful and know other people who would too, I'd really appreciate if you'd forward it to them 🙏

Thanks for reading!