Mechanizing the Methodology: How to find vulnerabilities while you're doing other things

Daniel Miessler, Founder, Unsupervised Learning twitter, linkedin
Red Team Village, DEF CON. August 8th, 2020.

You can also check out Daniel’s excellent blog at danielmiessler.com.

In this talk, Daniel Miessler discusses:

• The philosophy behind how he automates his recon and OSINT

• Several concrete examples of useful automations he’s built

• How these individual automation building blocks can be combined into powerful and complex chains

• How to set up lightweight, continuous scanning so you can focus your manual time on interesting targets

• Where to learn more

Philosophy / Methodology

The core philosophy behind Daniel’s automation approach is the Unix Philosophy:

Rather than creating a complex, comprehensive tool that does many things, instead, Daniel creates a series of small, self-contained scripts that do one thing well, for example, finding the subdomains of a given domain.

Each utility is designed to be composable: its output can be easily fed into another utility as input. Input formats are generally simple, text files with one IP address, domain name, etc. per line.

Another interested aspect of Daniel’s approach is that each utility is designed to answer a question; for example, “Which of these IPs are running web servers?”

But what should these individual scripts look like?

There are a couple of primary approaches one can take, and each has fundamental tradeoffs.

Frameworks (e.g. Amass, Intrigue, Spiderfoot)

• Pros: These are very powerful and can automate a number of steps and tasks for you, saving you time.

• Cons: They abstract steps from you so it can be unclear how they’re doing certain tasks.

  • This is important not just for your own personal learning, but there are often subtle tradeoffs in different ways to accomplish the same task (e.g. in speed, accuracy, or completeness), and when you don’t know how your framework is accomplishing what you ask it, you have blind spots.

Custom (writing custom code from scratch)

• Pros: You know exactly what your tools are doing and how. No magic.

• Cons: Doing this takes a massive amount of upfront time and effort.

Hybrid (leverage existing utilities, but wrap and organize them in a custom way)

• Pros: Highly customized for your workflow and you’re building a solid foundation - you know all the tradeoffs being made under the hood.

• Cons: Takes a bit of expertise and custom glue.

Recon Building Block Examples

After describing the thought process and philosophy behind his approach, Daniel gives a number of concrete examples.

Chaining Modules Together

Alright, so far we’ve built some interesting standalone primitives, building blocks that each answer an interesting question about a target.

The true power of this approach comes from combining these utilities to answer complex questions.

Now turn this idea up to 11: for example, you could have one module whose output feeds 10 other modules.

Continuous Monitoring

There are fancy ways to do automation, but you can also just use cron, a simple utility available on any Linux box.

And you can not just automate the scanning, but also automatically send yourself alerts (e.g. over email or Slack) when something interesting has been discovered. You can even have separate Slack channels dedicated to your favorite targets.

curl -X POST -H 'Content-type: application/json' \
  --data '{"text": "New open PostgresDB @ 1.2.3...4"}' WEBHOOK_URL 

Note: you’ll want to make sure that source modules have completed by the time the modules that consume their output are started.

Deployment

Manually making sure your Linux scanning box (don’t run from your personal computer) has the right tools and dependencies can be a pain.

Instead, we want to leverage modern infrastructure as code practices to define the tools and dependencies our scanning box needs. Then, we can make local updates, apply them, and be up and running in a few minutes.

Daniel uses Terraform and Ansible to deploy boxes to AWS, but check out Ben Bidmead’s Axiom for a slick way to deploy a Linux stack to Digital Ocean.

Continuous Improvement: Learn, Adjust, Repeat

One thing powerful about this approach is we’re codifying our domain knowledge in tools. We can then easily leverage these forever going forward.

How often do you watch a conference talk and think, “Oh nice, that’s neat. I should use that some day…” only to then forget whatever the clever trick was a few weeks later?

With this approach, when you come across something new and helpful, you can codify that into a module. It becomes another tool in your toolbelt, making you ever more effective and leveraged over time.

For keeping up with the latest tips and tactics, Daniel recommends following:

• Jason Haddix

• Ben Sadeghipour

• Tom Hudson

• Michael Skelton

• Luke Stephens

• Stok

• Jeff Foley

• Naffy

• Ben Bidmead

• The Cyber Mentor

You can also keep up with Daniel’s writing and research on his blog: danielmiessler.com.

Takeaways

And finally, Daniel ties it all together nicely: