AI and Machine Learning in Cybersecurity
An overview of current applications of AI/ML to cybersecurity with relevant links and a vision of where things are headed.
Clint Gibler
July 05, 2023
Ever since ChatGPT (and arguably before), the pace of innovation in machine learning / AI has been quite impressive.
This post is my attempt to:
Consolidate the many recent applications of AI to cybersecurity in one place
Tie together trends
And reason about where things may be headed.
Given the pace at which things are moving, this will likely be an incomplete list.
If there are meaningful blog posts, talks, or tools I should be aware of, please reach out and let me know! 🙏
Changelog
2023/07/03 - Wrote section on current applications/future work on AI in reverse engineering and added a number of relevant tools to the References section.
Themes and Where Things are Headed
Reverse Engineering
Current Applications
At the time of this writing, applications of AI in reverse engineering seem to fall into a very finite set of buckets.
Nearly all tools leverage LLMs to aid in program understanding:
Generate a plain text description of this {assembly or decompiled code}
Help me rename this {variable or function}
And less frequently:
Audit this code for vulnerabilities
Validate another tool’s security findings (Semgrep, in Callisto’s case)
See the “Reverse Engineering” subsection in the References section for a summary of ~8 tools you can review.
Future Work
Here are a few applications I find promising that do not yet seem to be explored.
Lower the barrier to entry for complex tools
IDA Pro, Ghidra, Binary Ninja, and the like are awesome, powerful tools, but they can be complex to learn how to use and have a high learning curve.
Instead of having to navigate many menus or read separate documentation, what if all of that info was indexed and there was helpful chat box in which you could simply ask, “How do I do ?” and it would give you a response right there, without interrupting your flow.
Personal Assistant
What if your tool was observing your behavior over time, after reversing many samples, and if you get stuck, perhaps offered advice like, “Last time you were in {situation}, you did {actions}, what if you tried that?”
And perhaps even automatically takes those actions for you, especially if, for example, there are a set of things you always do when starting a new reversing project.
Personal Expert Mentor
Building on the personal assistant idea, what if there was a model trained on the reversing behaviors and workflows of the best reversers in the world?
And as you were reversing, you’d see a little pop-up, kind of like auto-complete in GMail or a Hacker Clippy, that’d recommend (and maybe even do for you) what an expert would do.
You could imagine a solid implementation of this enabling junior reversers to perform at an intermediate level much more quickly, and maybe even make intermediate or advanced reversers even more effective.
An ideal implementation would not just say, “Do this,” but rather additionally provide the reasoning and context, the why behind it, to upskill the user’s thought processes, not just actions.
Lastly, I’m not sure if this would be as useful as the above, but it would be interesting for the expert bot to even get down to as specific as understanding a particular reverser’s style, like “In this situation, Rolf Rolles would do {this} or Malware Unicorn or Azeria would do {that}.”
Perhaps this could even fund a creator-economy / expert-model-as-a-service income stream for skilled reversers where they could bundle that knowledge and expertise and license the model, which individuals or companies could subscribe to.
Index and make available relevant context
There are a number of sources of information that could be indexed (for example, in a vector DB) and made available to a local model in your reversing platform of choice that would save you time providing context and preempting duplicate work.
For example:
Hashes, instruction sequences, and other IoCs of known malware, from VirusTotal, company blog write-ups, threat intel companies, and more.
This would make it easy to know if you're reversing something similar or identical to known malware, so rather than reversing from scratch, you could leverage (and maybe programmatically import) the work others have already done.
Indicators for common packers and obfuscators, as well as how to handle them.
Sharing proprietary data from your colleagues who may have done similar work.
Indexing known/popular functions or libraries so you’re wasting time reversing “known” functionality.
If you have feedback on any of these ideas or have other ideas I haven’t listed, I’d love to hear them! Happy to add them to this list if you’d like and credit you.
References
Web Security
Using LLMs to reverse JavaScript variable name minification
Jesse Luoto writes about how to reverse minified JavaScript using LLMs like ChatGPT and llama2 while keeping the code semantically intact.
Unminify and prettify the code.
Ask the LLM to describe the intent and a better name for variables.
Use Babel to do the renaming, which can effectively rename a JavaScript variable within its scope by operating on the code's Abstract Syntax Tree (AST), preserving how the code works.
How to automate API Specifications for Continuous Security Testing
Escape’s Younes Haddou describes a project in automatically generating an OpenAPI specification from source code, using Semgrep to extract routes and parameters and an LLM to infer the types of parameters.
tenable/Burp-extension-for-GPT
A Burp Suite extension that leverages OpenAI to analyze HTTP traffic and identify potential security concerns. “We tested some real-world scenarios and we found that GPT3.5 and GPT4 successfully identified XSS and misconfigured HTTP headers without requiring any additional fine-tuning.”
aress31/burpgpt
A Burp Suite extension that integrates OpenAI's GPT to perform an additional passive scan on web traffic observed. It also allows you to provide customizable prompts. This repo is no longer maintained, as the creator has launched a commercial version, which also supports local models.
AppSec
AI & Cybersecurity: Learnings from three months of Semgrep Assistant
Bence Nagy shares learnings from the private beta of GPT-4-powered Semgrep Assistant, which so far has received 95% positive feedback on its recommendations. Assistant is now in public beta, so if you use GitHub.com, you can now start using it immediately for free.
OX-GPT
OX Security’s Yael Citro describes OX-GPT, which can describe the impact of security issues, how to solve it, and recommend code that fixes it. They describe it as “AppSec’s First ChatGPT Integration,” despite Semgrep launching basically the same thing a month earlier.
Secure your machine learning with Semgrep
To strengthen the machine learning ecosystem, Trail of Bits’s Suha Hussain has analyzed the source code of libraries and identified some common problematic patterns. They have turned those patterns into 11 newly released Semgrep rules so it’s easy to find and fix those potential bugs.
Cracking Random Number Generators using Machine Learning – Part 1: xorshift128
NCC Group’s Mostafa Hassan shows how machine learning can predict the sequence of xorshift128’s random numbers using previously generated numbers without the knowledge of the seed. In part 2, he tackles Mersenne Twister.
Penetration Testing
GreyDGL/PentestGPT
A GPT-empowered penetration testing tool, by Gelei Deng, Víctor Mayoral Vilches, et al.
Attack Surface Manafement
projectdiscovery/openrisk
Tool by Project Discovery that reads nuclei (an OSS vulnerability scanner) output and generates a risk score for the host using GPT-3.
Code Review
Editor's note: I’ve seen a number of blog posts where people use LLMs for auditing source code. Some of them have pretty poor methodologies. The Trail of Bits post below is probably the best one I've seen so far in teasing out nuances.
I think there's a lot of improvements and potential future work in this space, I just want to point out the challenges and nuances so next time you see a blog post or company like, "ZOMG LLMs can find every vulnerability evar!" you'll think critically about their approach and methodology.
Using AI to find software vulnerabilities in XNU
Tim Michaud shares his experience of using a codebert-c model (an open source model from HuggingFace) trained with the codeparrot/github-code-clean dataset to pinpoint a memory corruption vulnerability affecting XNU.
Also, TIL about Fill-Mask: mask some of the words in a sentence and predict which words should replace those masks.
In the current state of the AIs used (Which could VERY well be due to my misuse of them), I did not find this a compelling use case. Perhaps if I understood things better and had the time/patience/more effort to put into this it could've been better (And likely would be!).
Codex (and GPT-4) can’t beat humans on smart contract audits
Trail of Bits' Artem Dinaburg, Josselin Feist, and Riccardo Schirone discuss their initial investigations into using LLMs for security auditing Solidity code. In short, it's not great at it today and the tooling is nascent.
There is no particularly good way to determine if one prompt is better than another or if you’re on the right track. Similarly, when a prompt fails against an input, it is frustratingly difficult to figure out why and to determine, programmatically, which prompts are merely returning the wrong result versus completely hallucinating and misbehaving.
Unit tests are also problematic; the results are not guaranteed to be the same across runs, and newer models may not provide the same results as prior ones. There is certainly a solution here, but again, the tooling developers expect just wasn’t present.
And a 🌶️ take:
people simping over AI based audits are going to lose their heads when they discover static analysis
— Benjamin Samuels (@thebensams)
Jun 4, 2023
Measuring LLMs ability to find security issues in code
GP Sandhu scans the OWASP Java Benchmark for vulnerabilities with both GPT-3.5 and GPT-4, measuring the true positive, false positive, and false negative rates, and released a repo with the results. Both models did fairly well finding true positives (especially GPT-4), but they both suffered from high false positives rates. GP also points out that this can get expensive: $5 for GPT-3.5 and ~$100 for GPT-4 to scan 2,740 Java files.
Lost in ChatGPT’s memories: escaping ChatGPT-3.5 memory issues to write CVE PoCs
Altin delves into ChatGPT’s memory limitations, offering solutions to escape the 4096-token limit, and outlining how to use ChatGPT as an assistant to analyze large codebases and write a CVE PoC for a resource exhaustion vulnerability discovered in Go’s textproto package.
The ChatGPT plugin OpenAI was using to assess the security of other plugins
rez0 shares the prompt, which was broken down into instructions, facts, and OpenAI’s policy.
Security Code Review With ChatGPT
NCC Group’s Chris Anley walks through having ChatGPT explain several vulnerabilities in the ‘Damn Vulnerable Web Application’ (DVWA) and analyzes its results.
TL;DR: Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does), and there are some other major problems that make it impractical. Also, both the CEO of OpenAI and ChatGPT itself say that you shouldn’t.
Overview Articles
How foundation models reshape cybersecurity tooling
Innovation Endeavors’ Harpi Singh and Dhruv Iyer do a nice round-up of current applications of LLMs to cybersecurity (search, code writing, vulnerability explanation, incident response and threat intelligence) across a number of vendors, and discuss promising opportunities: penetration testing, security reviews, and security-as-code generation.
Large language models in security
Innovation Endeavors’s Davis Treybig discusses a number of areas where he believes LLMs can be applied or integrated within security: SIEM, SAST, email security and phishing, SOAR, data loss prevention and data security, remediation, translating between security and everyone else, and vendor security questionnaires and compliance.
tenable/awesome-llm-cybersecurity-tools
A curated list of cybersecurity tools that leverage LLMs, by Tenable’s Olivia Fraser and Blake Kizer, across reverse engineering, network analysis, cloud security, and PoCs.
Can Generative AI Improve Your Cybersecurity Posture in 2023 and Beyond?
Mark Lynd provides an overview of some ways that AI can be applied to security, and lists a number of security vendors that have already integrated AI in some way into their products.
Limmen/awesome-rl-for-cybersecurity
A curated list of resources dedicated to reinforcement learning applied to cyber security, by Kim Hammar.
Cloud Security
AWS Docs GPT
AI-powered search & chat for AWS documentation.
LeanerCloud/ChatGPT-cloud-plugin
A ChatGPT plugin by Cristian Magherusan-Stanciu that will (one day) allow you to list and manage cloud resources from ChatGPT.
tenable/EscalateGPT
An AI-powered tool for discovering privilege escalation opportunities in AWS IAM configurations, by Tenable. “In our testing against real-world AWS environments, we found that GPT4 managed to identify complex scenarios of privilege escalation based on non-trivial policies through multi-IAM accounts.”
Container Security
robusta-dev/chatgpt-yaml-generator
By Robusta.dev: A ChatGPT plugin that gives ChatGPT up-to-date knowledge of every Kubernetes resource and their complete specs, lets ChatGPT query the Kubernetes OpenAPI schema to fetch resource definitions, and provides ChatGPT with schema validation capabilities.
Phishing / Disinformation
WormGPT – The Generative AI Tool Cybercriminals Are Using to Launch Business Email Compromise Attacks
Daniel Kelley covers a new cybercrime tool that enables more effective phishing emails by leveraging LLMs, cybercrime forum discussions around “jailbreaks,” and more.
Wasting Scam Callers’ Time with AI
A service that leverages GPT-4 to waste the time of telemarketers and scammers. The audio samples are pretty hilarious.
China is getting better at using AI to generate content designed to go viral in U.S. and other democracies. MS threat intel says China has “successfully engaged target audiences in at least 40 languages and grown its audience to over 103 million.”
CICERO: An AI agent that negotiates, persuades, and cooperates with people
Meta announces CICERO, the first AI to achieve human-level performance in the popular strategy game Diplomacy*. CICERO demonstrated this by playing on webDiplomacy.net, an online version of the game, where CICERO achieved more than double the average score of the human players and ranked in the top 10 percent of participants who played more than one game.
Diplomacy has been viewed for decades as a near-impossible grand challenge in AI because it requires players to master the art of understanding other people’s motivations and perspectives; make complex plans and adjust strategies; and then use natural language to reach agreements with other people, convince them to form partnerships and alliances, and more. CICERO is so effective at using natural language to negotiate with people in Diplomacy that they often favored working with CICERO over other human participants.
CICERO can deduce, for example, that later in the game it will need the support of one particular player, and then craft a strategy to win that person’s favor – and even recognize the risks and opportunities that that player sees from their particular point of view.
It is important to recognize that CICERO also sometimes generates inconsistent dialogue that can undermine its objectives.
While CICERO is only capable of playing Diplomacy, the technology behind this achievement is relevant to many real world applications. Controlling natural language generation via planning and RL, could, for example, ease communication barriers between humans and AI-powered agents. For instance, today’s AI assistants excel at simple question-answering tasks, like telling you the weather, but what if they could maintain a long-term conversation with the goal of teaching you a new skill?
Malicious Dependencies / Supply Chain Security
LLM-assisted Malware Review: AI and Humans Join Forces to Combat Malware
Endor Labs’ Henrik Plate describes some experiments they performed that suggest that LLM-based malware reviews can complement, but not yet substitute for human reviews.
1800 artifacts from PyPi and npm → 34 flagged as malware, 19/34 true positives.
Signals like in the Backstabber’s Knife Collection used.
“False-positives are predominantly due to incomplete and syntactically incorrect code snippets, which commonly happens if the prompt’s length restriction prevents us from uploading an entire file.”
“GPT can be tricked with help of several simple techniques to change an assessment from malicious to benign, e.g., by using innocent function names, including comments that indicate benign functionality or through inclusion of string literals.”
Reviewing Malware with LLMs: OpenAI vs. Vertex AI
Follow-up post in which Henrik makes several improvements and also tests Google’s Vertex AI.
Analysis Improvements
The removal of comments in suspicious code snippets (using Pygments) reduced exposure to prompt injection.
Asked for 0-9 risk score instead of binary classification.
Increased the context size, which also benefits from comment removals.
>90% of the time the two models rated within 1 point of the same score.
GPT-4 outperforms the other models for non-obfuscated code - better risk ratings and source code explanations.
Introducing Socket AI – ChatGPT-Powered Threat Analysis
Socket’s Mikola Lysenko describes how they’re using LLMs to find malicious packages or potentially dangerous/vulnerable npm and PyPi packages. He gives examples of information exfiltration, injection vulnerabilities, exposed credentials, backdoors, code that installs dependencies, prompt poisoning, etc. Here’s their earlier post on the same topic.
Self-enhancing pattern detection with LLMs: Our answer to uncovering malicious packages at scale
Apiiro’s Eli Shalom and Gil David describe how they combine a representation of a target package’s logic, capability analysis (e.g. does the package write files, perform HTTP requests, …), comparison to known malicious packages, clustering, and more to detect malicious packages at scale. Examples of malicious PyPi packages found.
Can you trust ChatGPT’s package recommendations?
Vulcan’s Bar Lanyado describes how an attacker can typosquat library names that ChatGPT is likely to hallucinate so that when developers use code recommended by the LLM they install the attacker’s malicious package instead.
Threat Modeling
Threat Modeling Example with ChatGPT
Rusty Newton walks through using ChatGPT to describe an example system architecture, the data flow between components, generate a list of threats using STRIDE, and then make recommendations on mitigating those threats.
More on GPT-3 and threat modeling
Adam Shostack asks a series of prompts about Kubernetes-related threats. Overall I don’t feel like it did a very good job at making them Kubernetes-specific.
Leveraging LLMs for Threat Modeling - GPT-3.5 vs Claude 2 vs GPT-4
Marcin Niemiec provided a fictitious high level project description, architecture description, and user story to the three models and evaluated how well they threat modeled. tl;dr: GPT-4 performed the best, you can review all of their outputs, and he created a GitHub Action to auto-threat model/security review.
DiagramGPT
(Not directly TM-ing, but related) Paste in a schema, infrastructure definition, or code snippet, or describe your diagram in plain language –> generate diagram.
Blue Team
Model-Assisted Threat Hunting (M-ATH) with the PEAK Framework
Splunk’s Ryan Fetterman continues his series on the PEAK Threat Hunting Framework. In this article, he explores the Model-Assisted Threat Hunting (M-ATH) technique, which leverages an algorithmic-driven approach to uncover potential leads for threat hunting. By streamlining or consolidating data, this method paves the way for more traditional hunting techniques.
Google brings generative AI to cybersecurity
Google announces Cloud Security AI Workbench, a cybersecurity suite powered by a specialized “security” AI language model called Sec-PaLM. Applications:
Mandiant’s Threat Intelligence AI, which will leverage Sec-PaLM to find, summarize and act on security threats.
VirusTotal: use Sec-PaLM to help subscribers analyze and explain the behavior of malicious scripts.
Chronicle search security events and interact “conversationally” with the results.
Security Command Center AI: “human-readable” explanations of attack exposure, including impacted assets, recommended mitigations and risk summaries for security, compliance and privacy findings.
We present That Escalated Quickly (TEQ), a machine learning framework that reduces alert fatigue with minimal changes to SOC workflows by predicting alert-level and incident-level actionability. On real-world data, the system is able to reduce the time it takes to respond to actionable incidents by 22.9%, suppress 54% of false positives with a 95.1% detection rate, and reduce the number of alerts an analyst needs to investigate within singular incidents by 14%.
Our TRAM Large Language Model Automates TTP Identification in CTI Reports
MITRE’s James Ross and Jackie Lasky describe how they’ve extended TRAM (Threat Report ATT&CK Mapper), a tool to automatically identify adversary tactics, techniques, and procedures (TTPs) in cyber threat intelligence (CTI) reports, to use LLMs— specifically, by fine-tuning SciBERT on 50 ATT&CK techniques.
mrwadams/attackgen
By Santander’s Matthew Adams: A tool that leverages LLMs and MITRE ATT&CK to generate tailored incident response scenarios based on user-selected threat actor groups and your organization's details.
Red Team
jiep/offensive-ai-compilation
Great list of useful resources on attacking AI models and using AI for offensive purposes (pentesting, malware, OSINT, phishing, generative AI, etc.) by José Escribano and Miguel Boza.
Fuzzing
AI-Powered Fuzzing: Breaking the Bug Hunting Barrier
Google’s Dongge Liu, Jonathan Metzman, and Oliver Chang describe how they were able to use an LLM to automatically write new fuzz targets for projects already being fuzzed by OSS-Fuzz, leading to greater code coverage and moar bugs!
Writing fuzzing test harnesses tends to be very manual/time intensive and a limiting factor in fuzzing effectiveness, so this could be a big deal if it works at scale.
CI Spark: LLM-Powered AI-Assistant for Creating Tests 15x Faster
Code Intelligence’s Khaled Yakdan describes CI Spark, a new product feature that leverages LLMs (optionally augmented by existing test code) to auto-generate test cases so their fuzzer gets higher code coverage. It can automatically identify fuzzing candidates (public functions that can be entry points) and generate code in JS/TypeScript, Java, and C/C++.
Content Moderation
Using GPT-4 for content moderation - By including a detailed policy in your prompt.
Reverse Engineering
Down the Rabbit Hole: Unusual Applications of OpenAI in Cybersecurity Tooling
Eugene Lim discusses his experiments with using OpenAI not just for human-based attacks like phishing and misinformation, specifically: reverse engineering assembly, analyzing Metasploit payloads, code reviews (e.g finding XSS), etc.
moyix/gpt-wpre:
Tool by Brendan Dolan-Gavitt that utilizes the text-davinci-003 model to generate a summary of an entire binary using decompiled code from Ghidra. It employs a recursive approach to create natural language summaries of a function's dependencies, providing them as contextual information for the function itself in an attempt to get around token limitations.
JusticeRage/Gepetto
A Python script by Ivan Kwiatkowski which uses OpenAI's gpt-3.5-turbo and GPT-4 models to provide meaning to functions decompiled by IDA Pro and rename variables.
MayerDaniel/ida_gpt
IDAPython script by Daniel Mayer that uses the unofficial ChatGPT API to generate a plain-text description of a targeted routine. The script then leverages ChatGPT again to obtain suggestions for variable and function names.
ant4g0nist/polar
A LLDB plugin that queries OpenAI's davinci-003 language model to explain the disassembly, by Chaitanya.
G-3PO: A Protocol Droid for Ghidra
Tenable's Olivia Lucca discusses using large language models to assist reverse engineers and introduces G-3PO, a Ghidra script that offers high-level explanatory comments for decompiled function code.
trailofbits/Codex-Decompiler:
A Ghidra plugin by Akshat Parikh that utilizes OpenAI's models to improve the decompilation and reverse engineering experience. It can:
Take the disassembly from Ghidra and feed it to OpenAI's codex model to decompile the code
Attempt to find vulnerabilities, generate a description, or decompile the Ghidra pseudocode using OpenAI
mahaloz/DAILA:
A decompiler-unified plugin by Zion Basque that leverages the OpenAI API to enhance your decompilation process by offering function identification, function summarisation and vulnerability detection. The plugin currently supports IDA, Binja and Ghidra.
JetP1ane/Callisto:
Callisto is an automated binary vulnerability analysis tool created by James B. It utilizes Ghidra for decompiling the binary, Semgrep for scanning the resulting C code, and GPT-3.5-Turbo to validate Semgrep's findings and potentially identify additional vulnerabilities.
Who will AI help more, attackers or Defenders?
Who Will AI Help More—Attackers or Defenders?
Daniel Miessler currently believes AI will most help attackers first, but it will help defenders more in the long term.
AI will help attackers automate and scale attacks in the short term before defenders have sufficient understanding of their environments, but in the long term, AI + a company’s context will give defenders the advantage.
How AI will affect cybersecurity: What we told the CFTC
Trail of Bits’ Dan Guido believes AI “has the potential to fundamentally change the balance between cyber offense and defense.”
Can a mid-level practitioner operate at an expert level with the help of AI? Our experience suggests yes.
AI is better for problems that require breadth of knowledge and where mistakes are acceptable (e.g., document this function, write a phishing email). It’s not great at problems that require mastery and correctness (e.g., find and exploit this iOS 0-day).
Bug bounties, phishing defenses, antivirus, IDS, and attribution will be among the first fields impacted as AI confers a greater advantage to attackers in the near term.
Feedback
Thanks for reading to the end!
If there are other topics you’d like me to add to this list, please reach out, I’d love to hear from you 😃