How to Create, Use, and Access Kubernetes Secrets
Kubernetes Secrets are secure objects that contain secret data such as logins, passwords, tokens, or keys.
We are taking the mystery out of Kubernetes Secrets. Read the following blog to find out what a Kubernetes Secret is, as well as the process of creating, using, and accessing them.
What are Kubernetes Secrets?
Kubernetes Secrets are secure objects that contain secret data such as logins, passwords, tokens, or keys.
There are many times when a Kubernetes Pod needs to use sensitive data. Think for examples of:
- SSH keys.
- Database passwords.
- OAuth tokens.
- API keys.
- Image registry keys.
Kubernetes is designed to have a declarative syntax. Object definitions are stored in YAML (or JSON) files and - typically - placed under version control. Adding confidential information to a version-controlled file (that anyone can view) is against any security best practice. For that reason, Kubernetes includes Secrets.
A Secret is just another Kubernetes object that stores restricted data to be used without being revealed. Kubernetes users can create Secrets, and the system establishes and uses Secrets.
Secrets are referenced through a file attached to the pod through a volume. The kubelet also uses Secrets when it needs to pull an image from an Image Registry that requires authentication (for example, a private Docker Hub account, AWS ECR, or Google GCR). Additionally, Kubernetes uses Secrets internally to enable Pods to access and communicate with the API server component. The system automatically manages API tokens through Secrets attached to the Pods.
Ways to Create Kubernetes Secrets Objects
There are several ways to create Secrets in Kubernetes. Your choice highly depends on the type of scenario.
Using The Command-Line and Text Files
Let’s have a quick example: you have a third-party service that you need your application to access. This service requires a username and password to authenticate requests. You need to create a Secret object that contains this information. Creating this secret using a file can be done as follows:
echo -n 'superuser' > ./username.txt
echo -n 'Q%FvqS$*F$k^6i' > ./password.txt
kubectl create secret generic app-user-cred --from-file=./username.txt --from-file=./password.txt
In this example, we used echo with the -n option to ensure that no newline character is added to our text (if we include a newline character it will be part of the username and password and authentication may fail). We created a file for the username and another for the password. Then we used kubectl to create a Secret object named “app-user-cred” that uses the data from the files we’ve just created.
Using The Command-Line and Literal Input
We could’ve equally generated the same password object without using files with a command like the following:
kubectl create secret generic app-user-cred --from-literal=username=devuser
--from-literal=password=Q\\%FvqS\\$\\*F\\$k\\^6i
However, notice that we had to add some extra characters to the password. Whenever the secret text contains special characters, we need to escape them by preceding each unique character with \\, which is only required when adding Secrets through the --from-literal option. Hence, it is much easier to use content from text files when defining Secrets to avoid confusion.
Using Definition Files
Like any other Kubernetes object, Secretes can be created using a definition file written in YAML or JSON.
Using Base64 and The Data Parameter
You can define the secret text as a base64-encoded string. For example, the following procedure creates and uses a YAML file to create a Secret to save our credentials:
$ echo -n 'superuser' | base64
c3VwZXJ1c2Vy
$ echo -n 'Q%FvqS$*F$k^6i' | base64
USVGdnFTJCpGJGteNmk=
Now that we have our credentials encoded in base64, let’s create the definition file mysecret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
superuser: YWRtaW4=
password: USVGdnFTJCpGJGteNmk=
Apply the file as usual using kubectl as follows:
kubectl apply -f mysecret.yaml
Using Clear Text and The StringData Parameter
You can skip encoding the strings and have Kubernetes does that for you automatically:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
superuser: 'superuser'
password: 'Q%FvqS$*F$k^6i'
Apply this definition similarly using kubectl:
kubectl apply -f mysecret.yaml
You can double check that Kubernetes did the base64 encoding for you by querying the secret data as follows:
$ kubectl get secret mysecret -o yaml
apiVersion: v1
data:
password: USVGdnFTJCpGJGteNmk=
superuser: c3VwZXJ1c2Vy
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},
"name":"mysecret","namespace":"default"},
"stringData":{"password":"Q%FvqS$*F$k^6i","superuser":"superuser"},"type":"Opaque"}
creationTimestamp: "2019-07-21T14:14:56Z"
name: mysecret
namespace: default
resourceVersion: "708718"
selfLink: /api/v1/namespaces/default/secrets/mysecret
uid: eaddc88c-abc1-11e9-8bff-025000000001
type: Opaque
Notice that you can use both stringData and data parameters when adding text to the Secret definition file. However, if you use stringData and data for defining the same parameter (for example, superuser), the stringData takes precedence.
Accessing the Kubernetes Secret Data
As depicted in the following image, the container accesses the secret through a volume in pretty much the same way as it does with configMap volumes:
You can get a Secret from Kubernetes the same way you get other objects using the get subcommand. For example:
$ kubectl get secret mysecret -o yaml
apiVersion: v1
data:
password: USVGdnFTJCpGJGteNmk=
superuser: c3VwZXJ1c2Vy
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},
"name":"mysecret","namespace":"default"},
"stringData":{"password":"Q%FvqS$*F$k^6i","superuser":"superuser"},"type":"Opaque"}
creationTimestamp: "2019-07-21T14:14:56Z"
name: mysecret
namespace: default
resourceVersion: "708718"
selfLink: /api/v1/namespaces/default/secrets/mysecret
uid: eaddc88c-abc1-11e9-8bff-025000000001
type: Opaque
Now, you can easily decode the base64-encoded strings as follows:
$ echo "superuser: $(base64 -D <<< 'c3VwZXJ1c2Vy')"
superuser: superuser
$ echo "password: $(base64 -D <<< 'USVGdnFTJCpGJGteNmk=')"
password: Q%FvqS$*F$k^6i
How to use Defined Secrets in Kubernetes
Now that you know how to define and decode a Secret, let’s see how we can use it for different purposes like authentication.
Accessing Secrets Through Files Inside a Volume
One way of using Secrets is to attach them to the Pod that needs them as volumes. If the Pod has more than one container, then each container must define a volume mount that points to the Secret volume. Each entry in a Secret is represented by a file. Let’s have an example:
---
apiVersion: v1
kind: Secret
metadata:
name: db-creds
type: Opaque
stringData:
user: 'root'
password: 'mypassword'
host: 'mydb.example.com'
---
apiVersion: v1
kind: Pod
metadata:
name: mysqlclient
spec:
containers:
- name: mysql
image: mysql
command: ["/bin/sh"]
args: ["-c","mysql -u `cat /mnt/db-creds/user)` -p`cat /mnt/db-creds/password)`
-h `cat /mnt/db-creds/host)`"]
volumeMounts:
- name: creds
mountPath: "/mnt/db-creds"
readOnly: true
volumes:
- name: creds
secret:
secretName: db-creds
The above YAML file includes a Secret, db-creds that defines the user, password, and host for a hypothetical database mydb.example.com. In the same file, we describe the Pod that will use this Secret. Setting all the related resources in the same file is common Kubernetes practice.
To access the Secret, the Pod has several important attributes:
- We define a volume that contains the Secret. The Secret name must match the one we set in the Secret definition.
- In the containers part, we define a volumeMounts entry for each container in the Pod. If you have more than one Secret, then each one has its volume and corresponding volume mount.
- The container should not have write access to the Secret volume. Hence, we set the readonly parameter to true.
The mountPath can point to any empty directory on the container’s filesystem. Any entries defined in the Secret are automatically converted to files (one file per entry) under this directory. The file is named after the key used to define the Secret.
Using the above configuration, we could access our remote database using the MySQL client command defined in the command and args part of the container. The command reads the contents of user, password, and host files found under the mounted path. For a real-world use case, this Pod can be deployed through a Kubernetes Job to execute SQL commands against the DB and gets terminated.
You can even override the default behavior of naming the Secret file after the name of the key by using .volumes[].secret.items array as follows:
volumes:
- name: creds
secret:
secretName: db-creds
items:
- key: user
path: db-username
- key: password
path: db-password
- key: host
path: db-host
Now you can refer to the files by their new names under the same mounted volume.
An important thing to notice here is that if you decide to use the .volumes[].secrets.secretName.items[] in a pod definition, then only the file paths defined in the items array will be used. If an entry was not set, it wouldn’t be accessible to the container, which can be a way to filter which secret listings are exposed to a container.
Accessing Secrets Through Environment Variables
Mounting volumes and using files that contain the Secret entries are more suited to scenarios where your application uses a file to authenticate itself to other applications. For example, key-based SSH authentication. But, for applications that use text on the command itself (like the mysql client used in our example), it is much easier and more convenient to use environment variables.
Let’s modify our Pod definition to access the Secret entries using environment variables instead of volumes. Notice that we won’t make any changes to the Secret definition.
---
apiVersion: v1
kind: Pod
metadata:
name: mysqlclient
spec:
containers:
- name: mysql
image: mysql
env:
- name: USER
valueFrom:
secretKeyRef:
name: db-creds
key: user
- name: PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
- name: HOST
valueFrom:
secretKeyRef:
name: db-creds
key: host
command: ["/bin/sh"]
args: ["-c","mysql -u $USER -p$PASSWORD -h $HOST"]
The changes we did here is removing the volumes and volumeMounts part and adding the env array to the container. The env array defines one or more entries that are visible to the container as environment variables. An env entry normally has a name (variable name) and the value that the variable holds. When getting its value from a secret, the env entry uses the valueFrom parameter, which searches for the Secret name and the Secret entry key through the secretKeyRef parameter.
Once all the Secret entries are exposed through environment variables to the container, they are accessed like any other environment variable. For example, $USER, $PASSWORD, and $HOST correspond to the user, password, and host Secret entries respectively.
Using ImagePullSecrets To Authenticate To Private Image Registries
Docker Hub and Google GCR are examples of image repositories that offer public as well as private image hosting. When you host your images that contain your proprietary application code, you usually choose to store those images privately. When you pull those private images on your local machine, you need to authenticate yourself first to the service. For example, Docker offers the docker login command that stores a file containing your credentials locally. This file is used whenever you need to pull or push an image to your repository. Google GCR offers a similar mechanism through its gcloud helper command. However, when Kubernetes pulls those images, the kubelet needs this file to be able to access the private registry. The imagePullSecrets parameter uses a Secret that holds the contents of the authentication file to enable the kubelet to pull the image.
Using ImagePullSecrets When You Don’t Have The Authentication File
You don’t need to create the authentication file first before injecting it into a Secret as this can be done in one step:
kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER
--docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
The above command automatically creates a secret that contains the necessary data required to authenticate to the image registry. Notice that although the command mentions “docker-registry” as a parameter, it can be equally used to authenticate Kubernetes to other image registries like GCR. Just make sure you follow the image-registry’s specific authentication information.
Now, once the Secret is created, we can use it to pull our private image like in the following example:
apiVersion: v1
kind: Pod
metadata:
name: myprivateapp
spec:
containers:
- name: myprivateapp
image: magalix/privateapp:v1
imagePullSecrets:
- name: myregistrykey
Using ImagePullSecrets When You Already Have The Authentication File
You may be already working with the private image registry and you just need Kubernetes to use your existing authentication file. You can create a Secret object that uses the contents of your authentication file as follows:
kubectl create secret generic myregistrykey --from-file=.dockerconfigjson=~/.docker/config.json
--type=kubernetes.io/dockerconfigjson
The above command looks similar to the ones we used earlier in this article to create Secrets from files. However, there are some required settings that cannot change:
- The file entry must have the entry name set to .dockerconfigjson
- The type must be set to kubernetes.io/dockerconfigjson
You can create this Secret through a definition file. However, instead of specifying the path to the authentication file, you provide its content in base64 encoding. For example:
apiVersion: v1
kind: Secret
metadata:
name: myregistrykey
namespace: myapps
data:
.dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbG
xsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxs
bGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ub
m5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type: kubernetes.io/dockerconfigjson
Kubernetes Secrets Summary
- Kubernetes Secrets are a way to store confidential information. Pods, the kubelet, and the system itself use them.
- Secrets can be created through the command line and definition files. When creating Secrets on the command line, you can point to existing files (and have their content stored), or directly enter the text. Any special characters must be escaped when entering the text directly.
- When using definition files, you can add the data in a base64 encoded format or plain text form. Kubernetes encodes the Secret data in base64 format. When you need to reveal a Secret text, you must base64-decode it.
- To enable containers to access Secrets, you can mount the Secret as a volume. All the Secret entries are represented as files under the mounted volume; each file contains the Secret content of the respective entry. You also have the option to pass them to the container as environment variables.
- The kubelet uses Secrets to pull images from private image registries that require authentication using the imagePullSecrets parameter. You can create the authentication file and the Secret in one step, or bring your authentication file and store it as a Secret.
Whitepaper: Production Ready Checklists for Kubernetes
Download these comprehensive checklists to help determine your internal readiness and gain an understanding of the areas you should update
