In part one of this series on Best Practices for Building and Designing Containers for Kubernetes, we looked at how to separate config from code in Kubernetes and why you need to do that. Specifically, we examined how config maps and environment variables are defined and configured. In this post, we continue that important discussion with a look at secrets management what it is and how you manage them.

Managing public configuration information in Kubernetes

Not all configuration information is safe to keep out in the “public” and many if not most Kubernetes hosted workloads need usernames/passwords, tokens, keys or other private information to securely connect to other services. There are a variety of options worth exploring here, each with its own set of positives and negatives.

Built-in Kubernetes secrets

Kubernetes provides a built in mechanism for storing configuration values that you would prefer to keep private. They can be access controlled to specific namespaces, and their contents are not shown by default in kubectl get or describe output. They are base64 encoded, so the contents are not immediately apparent even when you pull them directly from kubectl. Of course, base64 decoding is trivial.

These secrets are are stored in plaintext on the cluster’s etcd server, and unless etcd is configured to encrypt communication using TLS, are visible on the wire as the etcd cluster is synchronized. Furthermore anyone who has, or can gain root access to any node in the cluster can read all secrets data by impersonating the kubelet.

So, unless your security requirements are very low, it is recommended to use a third party solution to protect secret data.

There are three basic categories of secrets management solutions in Kubernetes which you should consider if your requirements go beyond the very low bar set by the built in Secrets functionality:

  1. Secrets management solutions from a cloud vendor
  2. Open source solutions you run yourself, either in the cluster or nearby
  3. Proprietary solutions from a variety of vendors

Cloud managed secret stores

If you are running in one of the major public clouds and already have an investment in its secrets management service, or you just want to get going quickly and aren’t concerned about potential vendor lock-in, Cloud managed solutions are a good bet.

Open source secrets managers

If you are on bare metal, want to avoid cloud vendor lock in, have concerns about the security of cloud vendor solutions, or need to integrate with an existing corporate standard you will probably need pick a software solution.

Given that there are very good open source solutions supported by Kubernetes vendors. Weaveworks Kubernetes Support covers both Vault and Sealed Secrets. I will not cover any of the proprietary solutions here.

Vault

The majority of my own personal experience is with open source solutions, and by far the most widely used, popular, and feature rich open source secrets manager used in Kubernetes is Vault.

Why Vault?

When we were deciding how to provide a consistent and feature rich secrets management solution for Kubernetes to our customers it was an obvious must have. Vault is more feature rich than the cloud managed soutions, and the one major argument that people have for cloud managed stores over vault is that it can be hard to setup and configure a performant HA Vault cluster, which is something we can alleviate with built in automation and support.

Vault Features

Most importantly for us, Vault is Cloud agnostic, secure, highly used and battle tested. Vault works perfectly with EKS, GKE, on prem clusters, and anywhere you might run kubernetes.

It also provides several unique features:

  • Completely private Cubbyholes where the token bearer is the only one who can access the data.
  • Dynamic secrets. Vault can automatically create accounts and credentials for you in databases and cloud IAMs.
  • PKI certificate and SSH certificate generation engines, which allow you to generate and store certificates with a single API call.
  • Cross region/Cross Cloud/Cross Datacenter replication with support for filters to restrict data that should not be transferred across clusters.
  • Support for a variety of authentication methods, and MFA where needed.

Sealed Secrets

One of the benefits of Kubernetes style orchestration is that configuration is based on a set of declarative json or yaml files, which can easily be stored in version control. Operational changes can be automated based on Git as a single point of truth. This means that every change to your workload configuration can be subject to the same pull request and peer review process as your application code.

However, traditional approaches to secrets management like Vault and all of the above cloud secrets stores introduce a second source of truth for secrets data, that is managed outside of Git— introducing another potential source of change/failure in your cluster that is tracked completely separately. This can complicate troubleshooting, create backdoors to peer review processes, and certainly acts to complicate audit logging for all cluster configuration changes.

Sealed Secrets was designed specifically to address this problem. It works by running a controller inside your Kubernetes cluster that takes secret data and a public key, and provides an encrypted string that you can use inside your standard configuration files, and can only be decrypted by the controller containing that private key.

This means that you can store secure credentials directly in configuration files in Git, share your Git repository with everyone who needs to access it, and yet none of those users has access to those credentials.

This can be used to create secure GitOps based workflows.

Because we believe there are massive advantages to a GitOps style workflow, the choice to support Sealed Secrets in Kubernetes offering was another obvious win.

Try out this tutorial, Managing Helm Releases the GitOps Way to see sealed secrets and GitOps in action.