- tl;dr sec
- Posts
- Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret?
Omer describes his quest to find a secrets management solution that supports GitOps workflows, is Kubernetes native, and has strong security properties, which lead to the development of a new tool, Kamus.
Clint Gibler
January 14, 2020
Omer describes his quest to find a secrets management solution that supports GitOps workflows, is Kubernetes native, and has strong security properties, which lead to the development of a new tool, Kamus.
Requirements
Going in to his search, Omer had the following requirements for an ideal solution:
GitOps (placing the code, manifest files, and secrets in a git repo that is then distributed to the Kubernetes pods.
Kubernetes native - easy, seamless integration with Kubernetes
Secure - High quality, secure code, and “one-way encryption” (once the secret is encrypted, it can only be decrypted by the application that will be using it)
The pod itself is considered out of scope for this talk, as there are a number of orthogonal concerns and security requirements (e.g. Who can “SSH” in? What is running on the pod? Does the code leak the secrets?)
Option #1: Kubermetes Secrets
At first, Kubernetes Secrets seemed like a quick, easy win. However, secrets are base64 encoded, which is called out in the risks section in the documentation.
If you configure the secret through a manifest (JSON or YAML) file which has the secret data encoded as base64, sharing this file or checking it in to a source repository means the secret is compromised. Base64 encoding is not an encryption method and is considered the same as plain text.
There are some other solutions, like helm-secrets and sealed-secrets, but using them has some key management challenges, you become coupled to a specific cluster/deployment method, and any change to the secret requires decryption.
Consuming can be done via environment variables, which has tradeoffs, or volume mounts or configuration files.
Option #2: Hashicorp Vault
Vault has a number of nice properties, but it requires a separate storage of secrets and deployment files (everything isn’t in a single git repo, so no single source of truth), requires an external permission model, and you have to manage the deployment and running of the service. There are some cloud vendor alternatives, such as AWS Secrets Manager, Azure Key Vault, and GCP berglas.
Option #3: Roll Your Own (Kamus)
Omer took inspiration from how Travis supports encrypted secrets, where users can include encrypted values in their travis.yml files that can only be read by Travis CI.
So, Omer and his colleagues built Kamus, which is basically Travis secret encryption for Kubernetes.
Demo
Omer gives a demo of using Kamus at 19:52, and you can also follow along with a Kalmus example in their GitHub repo.
Kamus Security
Kamus has the following permission model:
Protections
User: secure by default permission model and CLI (enforces HTTPS, has support for certificate pinning).
Git: strong encryption using Azure KeyVault/GCP KMS (supports HSM protection and IP filtering, the latter of which makes it tough to exfiltrate secrets even in the case of a successful attack), one-way encryption.
Pod: secure by default permission model, in-memory volume for decrypted files.
Kamus API: Separate pods (crypto component is never exposed to the Internet), the encryptor supports authentication (if you want to limit who can speak to it), and every commit is scanned by Checkmarx (SAST), ZAP (DAST), and Snyk (out of date dependencies).
Accepted Risks
Clear text traffic inside the cluster (in practice it’s very hard to man-in-the-middle Kubernetes traffic in the cloud)
Any pod in the same namespace can mount any service account (Pod impersonation)
Service account token never expires
References
Kamus is also discussed in this blog post, and you can find the source code here.