• tl;dr sec
  • Posts
  • [tl;dr sec] #339 - Hugging Face's Incident Report, Context Bombs, AI does Cryptanalysis

[tl;dr sec] #339 - Hugging Face's Incident Report, Context Bombs, AI does Cryptanalysis

HF does a detailed play-by-play of the incident with a neat visualization, stopping AI attackers via content that triggers their guardrails, Mythos finds attacks on HAWK and AES

Hey there,

I hope you’ve been doing well!

🏜️ Hacker Summer Camp

I’m excited about Black Hat and DEF CON next week!

It’s always such a delight to catch up with friends and meet new cool people. Even if it’s going to be 115°F, which is roughly 40 degrees too hot for me.

We’re doing some workshops with SpecterOps on “Operationalizing Codex for Malware Triage” and a talk (same page), we’ve got booths at Black Hat and DEF CON, and I’ll be around at various events.

I think OpenAI is having a happy hour, but I don’t have a link handy. I’ll try to post about it on my LinkedIn next week.

Oh yeah funny anecdote- apparently some of my colleagues have used Codex + ChatGPT Sites to spin up a quick web app tracking like over 1,000 Hacker Summer Camp events.

They then used Codex to mass apply to events, and are so far: shooting machine guns from a helicopter, driving sports cars, and more 😂 What a time to be alive.

Hope to see you next week! And if not, I hope you have an awesome time in Vegas, or perhaps you more wisely chose to stay home.

Sponsor

📣 Attackers are spoofing big brands to get your AI agent to install malware

Island Security Research uncovered 7,600 malicious GitHub repositories — over 800 posing as AI Skills or MCP servers — with some using Walmart, Databricks, Gmail, Salesforce, Spotify, and other big brand names to look legitimate. 

The payload: SmartLoader and the StealC infostealer, with 14M+ downloads. Worse, no one had to click a bad link. Claude Code, Gemini, and ChatGPT all discovered these fake capabilities on their own and treated the attacker's README as legitimate documentation. We call it AgentBaiting.

Very interesting: an AI agent searches for a new capability (e.g. Skill or MCP) → finds malicious GitHub repo → pwned. Oof 14M+ downloads is a lot. Nice details, great that they included IOCs 👍️ 

AppSec

Control Reliability Engineering (CRE): Applying SRE Principles to Cybersecurity Controls
Phil Venables proposes Control Reliability Engineering (CRE), an application of Site Reliability Engineering to security controls that treats a failed control with the same weight as a breach, since a broken control looks identical to a working one until something tests it. CRE lays out 10 elements to close that gap, control cataloging with a formal ontology, controls-as-code with CI/CD pipelines, Control SLIs/SLOs and error budgets, Continuous Control Monitoring backed by synthetic event injection to catch silent failures, Control Readiness Reviews before deployment, gradual rollout for changes, and blameless postmortems on failed controls. Instead of triaging alerts and running subjective risk assessments, security teams track control health directly through policy evaluation success rates and coverage ratios.

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident
Hugging Face published a forensic timeline of an intrusion by an autonomous AI agent running OpenAI's ExploitGym evaluation harness, reconstructing ~17,600 attacker actions across two stages. The agent first escaped OpenAI's sandbox via a zero-day in the package registry cache proxy. Then it breached Hugging Face's infrastructure through two injection vectors against the same production Kubernetes dataset loader, an HDF5 external raw storage exploit that leaked pod secrets and worker source code, and a Jinja2 template injection that achieved arbitrary code execution. Once inside, the agent escalated to cluster-admin across multiple internal clusters in under 13 hours, then abused a misconfigured internal service-connector whose single shared credential bound to system:masters gave it cluster-admin everywhere in one second, enrolled devices into HF's Tailscale mesh with a stolen auth key, and ran C2 entirely through public services with chunked+XOR+gzip encoding that naive log scans missed.

Claude Opus and Fable refused to analyze the attack logs due to cybersecurity guardrails (no mention of OpenAI models being used and getting refusals 👀), the team switched to open-source GLM-5.2 running on their own infrastructure.

💡 Very detailed write-up, great read. Especially the lateral movement and C2/exfil sections. HF’s interactive replay is well worth checking out. See also the updated OpenAI blog about the incident.

Sponsor

📣 The Practitioner's Playbook for Asset Intelligence

AI is scaling attacks, but they're still hitting the same weak spots: unpatched devices, over-permissioned identities, and configuration drift no one caught. The real fix isn't another tool, it's fixing the foundation. This playbook breaks down the five disciplines that separate teams who stay ahead of exposure from those who don't: Collection, Coverage, Context, Containment, and Cooperation. Get a quick win for each one you can put into practice this week.

The fundamentals like asset management are still, if not more critical than ever. I like the focus on quick wins.

Cloud Security

Introducing ClickOps Sentinel: AI Context for AWS Console Changes
Victor Grenu introduces ClickOps Sentinel, an open-source pipeline that detects AWS Console changes in near real-time by filtering EventBridge events for CloudTrail's sessionCredentialFromConsole flag and readOnly=false, and hands each change to a Claude agent on Amazon Bedrock for investigation. The agent retraces the whole console session via CloudTrail LookupEvents, checks adjacent activity from the same IP, consults long-term memory in Bedrock AgentCore Memory, and produces a verdict (manual-change-confirmed, likely-sanctioned-activity, suspicious) with confidence, security implications, FinOps impact, session narrative, and a remediation recommendation. Alerts land in Slack or Microsoft Teams via Amazon Q Developer chat, or as rich HTML emails through external providers like Resend or SendGrid, and the tool deploys via AWS SAM CLI.

Rapid7 Labs: Investigating Persistence Mechanisms in AWS
Rapid7's Jan Blažek describes four AWS persistence techniques attackers use to maintain access after initial compromise, providing CloudTrail-based LEQL queries for detection and investigation. The techniques include creating IAM users with follow-up credential and permission additions (iam:CreateAccessKey, iam:AttachUserPolicy), modifying assume role policies to allow external AWS accounts (iam:UpdateAssumeRolePolicy), deploying backdoored Lambda functions triggered via public URLs (lambda:CreateFunctionUrlConfig) or API Gateway integrations, and creating federated user sessions via sts:GetFederationToken whose temporary credentials survive access key rotation.

Jan warns that federated sessions require applying deny-all policies to the source IAM user until temporary credentials expire, since AWS persistence typically abuses legitimate identity and automation features rather than exploiting flaws, so effective detection requires pivoting from the initial alert to every related identity, credential, session, and policy.

Blue Team

manishrawat21/Cisa-KEV-Threat-Intel-Orchestrator
By Manish Rawat: An n8n workflow that turns newly weaponized CVEs from the CISA KEV catalog into Sigma detection rules every Monday, cutting 4-6 hours of manual work per CVE down to zero. The pipeline uses Google Gemini to generate rules targeting specific products with Sysmon EventIDs and MITRE ATT&CK mappings, logs each CVE to Google Sheets for compliance, and emails a weekly analyst briefing.

Context bombs: stopping AI attackers in their tracks
Tracebit demonstrates a canary token technique called "context bombs," (GitHub repo with payloads here) where decoy AWS Secrets Manager secrets contain content designed to trip an AI attacker's safety guardrails when the model reads them. Testing five frontier models (Opus 4.8, Gemini 3.1 Pro, GLM 5.2, DeepSeek V4-Pro, Kimi K2.6) across 152 attack runs in a simulated AWS cyber range, a single context bomb cut average admin-access from 57% to 5%, full compromise from 36% to 1%, and Opus 4.8 specifically from 93% baseline to 0%. Effectiveness depends on payload family, with sensitive biological content working against Western models while politically sensitive Chinese content works against Chinese-lab models. Every context bomb hit also fires a canary alert, giving defenders both detection and disruption in a single mechanism. This approach exploits the fact that AI agents must send retrieved data back to their LLM for analysis, creating a chokepoint where content filters can detect and block malicious activity.

💡 This is a very clever and fun idea- purposefully planting content that will trip an AI agent’s safety guardrails and thus stop it. The Western vs Chinese bomb differences is also interesting 👀 

Red Team

googleprojectzero/sandbox-attacksurface-analysis-tools
A set of tools to analyze Windows sandboxes for exposed attack surface, by Google Project Zero’s James Forshaw.

MatheuZSecurity/Furtex
By MatheuZ: A Linux post-exploitation and evasion research toolkit built around io_uring and eBPF, with 125+ tools across five categories. The core idea is that io_uring's kernel workqueue path sidesteps a large class of EDR hooks natively, since operations submitted through io_uring never fire syscall entry tracepoints. Additional tools handle what io_uring can't bypass on its own, disabling kprobes, BPF LSM hooks, Linux audit, and netfilter, while a dedicated Falco module targets all 25 of its default rules.

AI + Security

Quicklinks:

  • ethz-spylab/agentdojo - A dynamic benchmark environment for evaluating prompt injection attacks and defenses in LLM agents.

  •  Why detection architecture matters more than the AI model - AI changed email security, but even with modern models, SOCs still run into the same structural issues. Josh Kamdjou (CEO, Sublime) and Dmitri Alperovitch (Co-Founder, Crowdstrike) make the case for why architecture, not AI models, is what actually lets defenders keep pace.*

  • amElnagdy/guard-skills - Review prompts for AI coding agents that catch LLM failure modes, including clean-code-guard (catch-all error swallowing, hardcoded success returns, hallucinated APIs), test-guard (flags mock abuse and duplicate tests), docs-guard (verifies every documentation claim against the codebase), and wp-guard (WordPress security).

*Sponsored

Dicklesworthstone/destructive_command_guard
By Jeff Emanuel: a high-performance hook for AI coding agents (Claude Code, Codex CLI, Gemini CLI, …) that intercepts and blocks destructive commands before execution using a modular pack system, with 50+ security packs covering databases, Kubernetes, Docker, cloud providers, and more. The tool uses a three-tier heredoc/inline-script scanning architecture with SIMD-accelerated filtering and context-aware pattern matching that distinguishes executable code from data/comments.

It also includes a bounded failure policy with configurable fail-closed mode, agent-specific trust profiles, allowlist management, and scan mode for CI/pre-commit hooks with format-aware extractors (shell scripts, Dockerfiles, GitHub Actions, Makefiles, etc.).

💡 Making sure agents don’t take destructive seems to be one of the most common concerns I hear from security leaders right now.

We built a test corpus for AI agent egress security tools
Pipelab's Josh Waldrep introduces Agent Egress Bench, an open-source test corpus for security tools that sit between AI agents and the network, covering 197 logical cases across 18 categories including secret exfiltration, prompt injection, SSRF, hostname exfiltration, MCP tool poisoning and chain detection, MCP drift, A2A protocol scanning, WebSocket DLP, encoding evasion, shell obfuscation, and crypto/financial DLP.

Most benchmarks in this space test whether the LLM behaves correctly. Agent Egress Bench tests whether the firewall, proxy, or scanner sitting between the agent and the network catches the attack, with each case mapped to the OWASP Top 10 for Agentic Applications. To score results, an optional Gauntlet program reports four independent metrics: containment, false positive rate, detection, and evidence.

Some thoughts about Anthropic’s new cryptanalysis results
John Hopkins professor Matthew Green gives an overview and commentary on two cryptanalysis results from Anthropic's Claude Mythos: a key recovery attack on HAWK (a post-quantum signature scheme candidate) that roughly halves its security bits using existing techniques, and a modest improvement on 7-round AES attacks. What's notable about the HAWK result is that it doesn't invent new mathematics, but simply applies existing cryptanalytic tools more thoroughly (the type of work AI excels at), with the researchers achieving these results by essentially telling the model to grind away at problems without detailed human intervention.

Matthew argues this is an ideal time for AI cryptanalysis capabilities to emerge, as we're mid-transition to post-quantum algorithms, meaning AI can stress-test new standards before deployment.

Anthropic also partnered with academics at ETH Zurich, Tel Aviv University, and TU Berlin to build CryptanalysisBench, a benchmark that packages together many cryptographic ciphers and makes it easy for others to evaluate the capabilities of LLMs. The HAWK attack took about 60 hours and $100,000 in API costs.

💡 Super cool to see frontier models applied to cryptanalysis. Improving these underlying algorithms keeps us all a bit safer. It’s interesting to see how as models get better, you can get meaningful results with simpler/lower effort prompts, in this case writing a simple prompt to Claude that rewrote the agent harness that improved the analysis. The security researchers on my team at OpenAI have found the same thing.

Misc

✉️ Wrapping Up

Have questions, comments, or feedback? Just reply directly, I’d love to hear from you.

If you find this newsletter useful and know other people who would too, I'd really appreciate if you'd forward it to them 🙏

Thanks for reading!

Cheers,
Clint

P.S. Feel free to connect with me on LinkedIn 👋