Managing Secrets in Kubernetes
Learn how to separate code from configuration in Kubernetes and Kubernetes secrets management tips in this post.
In this post, we continue our discussion on how to separate code from configuration in Kubernetes. We also answer the questions “How secrets are managed in Kubernetes?” and “How do you keep secrets in Kubernetes?” by explaining the different ways you can manage Kubernetes secrets.
In part one of this series on Best Practices for Building and Designing Containers for Kubernetes, we looked at how to separate config from code in Kubernetes and why you need to do that. Specifically, we examined how config maps and environment variables are defined and configured. In this post, we continue that important discussion with a look at secrets management, what it is, and how you manage them.
Managing public configuration information in Kubernetes
Not all configuration information is safe to keep out in the “public” and many if not most Kubernetes hosted workloads need usernames/passwords, tokens, keys or other private information to securely connect to other services. There are a variety of options worth exploring here, each with its own set of positives and negatives.
Built-in Kubernetes secrets
Kubernetes secrets manager provides a built-in mechanism for storing configuration values that you would prefer to keep private. They can be access-controlled to specific namespaces, and their contents are not shown by default in kubectl get or describe output. They are base64 encoded, so the contents are not immediately apparent even when you pull them directly from kubectl. Of course, base64 decoding is trivial.
These secrets are stored in plaintext on the cluster’s etcd server, and unless etcd is configured to encrypt communication using TLS, are visible on the wire as the etcd cluster is synchronized. Furthermore, anyone who has, or can gain root access to any node in the cluster can read all secrets data by impersonating the kubelet.
So, unless your security requirements are very low, it is recommended to use a third-party solution to protect secret data.
There are three basic categories of secrets management solutions in Kubernetes which you should consider if your requirements go beyond the very low bar set by the built-in Secrets functionality:
Secrets management solutions from a cloud vendor
Open source solutions you run yourself, either in the cluster or nearby
Proprietary solutions from a variety of vendors
Cloud-managed secret stores
If you are running in one of the major public clouds and already have an investment in its secrets management service, or you just want to get going quickly and aren’t concerned about potential vendor lock-in, Cloud managed solutions are a good bet.
Open-source Kubernetes secrets managers
If you are on bare metal, want to avoid cloud vendor lock-in, have concerns about the security of cloud vendor solutions, or need to integrate with an existing corporate standard, you will probably need to pick a software solution.
Weaveworks Kubernetes Platform includes support for both Vault and Sealed Secrets. I will not cover any of the proprietary solutions here.
Vault
The majority of my personal experience is with open-source solutions, and by far, the most widely used, popular, and feature-rich open-source Kubernetes secrets manager is Vault.
Why Vault?
When we were deciding how to provide a consistent and feature-rich Kubernetes secrets management solution to our customers, Vault was an obvious must-have. Vault is more feature rich than the cloud-managed solutions, and the one major argument that people have for cloud-managed stores over vault is that it can be hard to setup and configure a performant HA Vault cluster, which is something we can alleviate with built-in automation and support.
Vault Features
Most importantly for us, Vault is Cloud agnostic, secure, highly used and battle-tested. Vault works perfectly with EKS, GKE, on prem clusters, and anywhere you might run kubernetes.
It also provides several unique features:
Completely private Cubbyholes where the token bearer is the only one who can access the data.
Dynamic secrets. Vault can automatically create accounts and credentials for you in databases and cloud IAMs.
PKI certificate and SSH certificate generation engines, which allow you to generate and store certificates with a single API call.
Cross region/Cross Cloud/Cross Datacenter replication with support for filters to restrict data that should not be transferred across clusters.
Support for a variety of authentication methods, and MFA where needed.
Sealed Secrets
One of the benefits of Kubernetes-style orchestration is that configuration is based on a set of declarative json or yaml files, which can easily be stored in version control. Operational changes can be automated based on Git as a single point of truth. This means that every change to your workload configuration can be subject to the same pull request and peer review process as your application code.
However, traditional approaches to secrets management like Vault and all of the above cloud secrets stores introduce a second source of truth for secrets data, that is managed outside of Git—introducing another potential source of change/failure in your cluster that is tracked completely separately. This can complicate troubleshooting, create backdoors to peer review processes, and certainly acts to complicate audit logging for all cluster configuration changes.
Sealed Secrets was designed specifically to address this problem. It works by running a controller inside your Kubernetes cluster that takes secret data and a public key, and provides an encrypted string that you can use inside your standard configuration files, and can only be decrypted by the controller containing that private key.
This means that you can store secure credentials directly in configuration files in Git, share your Git repository with everyone who needs to access it, and yet none of those users has access to those credentials.
This can be used to create secure GitOps-based workflows.
Because we believe there are massive advantages to a GitOps style workflow, the choice to support Sealed Secrets in Kubernetes offering was another obvious win.
Try out our Managing Helm Releases the GitOps Way tutorial to see sealed secrets and GitOps in action.
Production-Ready Kubernetes Guide: What It Means and How to Achieve It.
Find out what production ready means, the cultural changes you need to make on your team, as well as the most important requirements to consider when using and taking advantage of Kubernetes in production.
