• tl;dr sec
  • Posts
  • How to Lose a Container in 10 Minutes

How to Lose a Container in 10 Minutes

In this talk, Sarah discusses container and Kubernetes best practices, insecure defaults to watch out for, and what happens when you do everything wrong and make your container or cluster publicly available on the Internet.

Author

Clint Gibler
January 14, 2020

Sarah Young, Azure Security Architect, Microsoft twitter, linkedin
abstract slides video

Moving to the Cloud? Tidy House

If you’re making the move to the cloud and containerizing your application, use this as a opportunity to tidy it up.

  • Everything should be encrypted at rest and in transit.

  • Use TLS 1.2 or 1.3.

  • Remove deprecated protocols.

  • Tidy up the code and simplify where possible.

Use Only Trusted Base Images

Make sure you know where your container images come from.

Try to minimize your use of images from the Internet by keeping your own base images.

You can use a private image repository, such as those from cloud providers (GCP Container Registry, Azure Container Registry, AWS Elastic Container Registry), GitLab container registry, or even run a private DockerHub registry (see also Notary).

Kubernetes’ Insecure Defaults

Kubernetes default configurations generally are fairly insecure, and you need to work through the orchestrator configs to secure them.

Especially dangerous Kubernetes defaults include the API server listening on port 8080 (unauth) and “secrets management” using etcd.

The CIS Kubernetes benchmark has some good advice for securing Kubernetes defaults.

Secrets Management

Rather than baking credentials and secrets into containers, instead pass them in as environment variables.

Kubernetes stores secrets in etcd, encoded in base64. D’oh. Instead, use the built-in secrets management functionality provided by major cloud platforms, such as AWS Secrets Manager, Azure Key Vault, and GCP berglas. If those don’t work for your use cases, you could also use a third-party secrets management system like Vault.

Rotate your secrets regularly, at least once a month if you can. Talk with your risk team to determine what makes sense.

See Can Kubernetes Keep a Secret? for more thoughts on storing secrets in Kubernetes and the tradeoffs on passing in secrets via environment variables.

Container Privilege Best Practices

#1 thing: Don’t run as root.

If you must run as root (e.g. your container needs to modify the host system), use runtime security tools to limit what is accessible. Check out Aqua Enforcer, SELinux, AppArmor and seccomp profiles.

Orchestrator Privilege Best Practices

As mentioned previously, Kubernetes had (and still has) some insecure defaults, including:

  • Anonymous user access isn’t disabled.

  • The dashboard had full admin privileges by default (prior to v1.7).

  • No RBAC before v1.8.

If you don’t have the internal expertise to manage Kubernetes, there’s no shame in using one of the managed offerings cloud platforms provide.

There are also some open source tools to declaratively manage Kubernetes clusters, such as kube-aws.

Ensure Your Tools Work on Containers / Kubernetes

When you move to the cloud and/or are containerizing applications, don’t assume your same tools will work. They might not.

Most security tools need to be specifically container/k8s aware, or may need additional plugins. Review the following areas: IDS/heuristics, vulnerability scanning, your SIEM, runtime security, and auditing.

The same goes for your CI/CD pipeline - the tools you’re using may need to altered to work properly or replaced entirely.

Benchmark your tools with both developers and the security team. After all, a tool that doesn’t work for devs won’t get used.

Experiment: What Happens When You Do Everything Wrong?

Sarah spent a few months spinning up containers and Kubernetes clusters and leaving them open to the Internet. She intentionally disabled many built-in security features and overally did the inverse of everything she advocated doing in this talk.

She ran the containers on a cloud hosting provider so it wasn’t on her own infrastructure, and she paid through PayPal so she could avoid running up a big bill. The base containers she used were for Wordpress and Ubuntu 14.04.

Surprisingly, they were not compromised as quickly as she expected, and the overall level of attacks were less than you’d think. One potential reason for this is that the cloud hosting provider may be offering other protections transparently behind the scenes.

Resources