Kubernetes RBAC 101

November 18, 2019

For an overview, implementation information, and specific details regarding Kubernetes RBAC, check out this blog post!

Related posts

KubeCon EU 2023 Recap – GitOps Sessions on Flux with OCI, Liquid Metal CI/CD Platforms & Telco Cloud Platforms

Extending GitOps Beyond Kubernetes with Terraform Controller

Kubernetes On-Premise - What You Need to Know

Overview

RBAC is a security design that restricts access to valuable resources based on the role the user holds, hence the name role-based. To understand the importance and the need of having RBAC policies in place, let’s consider a system that doesn’t use it. Let’s say that you have an HR management solution, but the only security access measure used is that users must authenticate themselves through a username and a password. Having provided their credentials, users gain full access to every module in the system (recruitment, training, staff performance, salaries, etc.). A slightly more secure system will differentiate between regular user access and “admin” access, with the latter providing potentially destructive privileges. For example, ordinary users cannot delete a module from the system, whereas an administrator can. But still, users without admin access can read and modify the module’s data regardless of whether their current job entails doing this.

If you worked as a Linux administrator for any length of time, you appreciate the importance of having a security system that implements a security matrix of access and authority. In the old days of Linux and UNIX, you could either be a “normal” user with minimal access to the system resources, or you can have “root” access. Root access virtually gives you full control over the machine that you can accidentally bring the whole system down. Needless to say that if an intruder could gain access to this root account, your entire system is at high risk. Accordingly, RBAC systems were introduced.

In a system that uses RBAC, there is minimal mention of the “superuser” or the administrator who has access to everything. Instead, there’s more reference to the access level, the role, and the privilege. Even administrators can be categorized based on their job requirements. So, backup administrators should have full access to the tools that they use to do full, incremental, and differential backups. But they shouldn’t be able to stop the webserver or change the system’s date and time, for example.

Kubernetes Implementation Of RBAC

Now that you know what RBAC is and why it needs to be used in any computer system that has valuable resources let’s see how Kubernetes implements it. But before we can discuss RBAC in Kubernetes, we need to create a user for testing.

Creating a Kubernetes User Account Using X509 Client Certificate

If you have a Kubernetes cluster in place for testing purposes, you’re most probably the cluster owner or the administrator. Let’s create a user account for our labs. Kubernetes supports several user authentication methods. It also supports combining more than one to authenticate a user. If one of the chained methods fails, the user is not verified. In this example, we’ll use only one authentication method, the X509 certificate to create a user account called Magalix.

First, we need to create the client key (you’ll need OpenSSL command installed on your system):

openssl genrsa -out magalix.key 2048

Then, we need to create a certificate signing request:

openssl req -new -key magalix.key -out magalix.csr -subj "/CN=magalix"

Notice that the CN part of the subject must contain the username.

Since we have the user key and the signing request, we can use the cluster certificate and certificate key to sign this request:

openssl x509 -req -in magalix.csr -CA ca.crt -CAkey ca.key -CAcreateserial 
-out magalix.crt -days 300

Note For Local And Self-Hosted Clusters Users

For the above command to run, you need to have the cluster certificate (ca.crt) and the certificate key (ca.key). If you’re using minikube, you’ll find those files under ~/.minikube/. In my case, I was using Docker for Desktop on macOS. I had to use the following commands to get the certificate and the file:

kubectl cp kube-apiserver-docker-for-desktop:run/config/pki/ca.crt -n kube-system ca.crt
kubectl cp kube-apiserver-docker-for-desktop:run/config/pki/ca.key -n kube-system ca.key

For other self-hosted clusters, you can find the path to the certificate and the key under /etc/kubernetes/manifests/kube-apiserver.manifest.

Let’s add the user’s credentials to our kubeconfig file:

kubectl config set-credentials magalix  --client-certificate=magalix.crt 
--client-key=magalix.key

We’ll also create a context for this user and associate it with our cluster:

kubectl config set-context magalix-context --cluster=docker-for-desktop-cluster --user=magalix

If you have a look at the ~/.kube/config file you should see the new user data added to the file.

Let’s test what level of privilege this user has:

$ kubectl --user=magalix get pods
Error from server (Forbidden): pods is forbidden: 
User "magalix" cannot list pods in the namespace "default"

If you try the above command with other Kubernetes like get deployments or get jobs, you will find that the user does not have any privileges. We need to start giving it the necessary authorization. A good security practice is to grant users the access level that they need to execute their jobs only.

Authorization Using Kubernetes Roles

A Role in Kubernetes is as a Group in other RBAC implementations. Instead of defining different authorization rules for each user, you attach those rules to a group and add users to it. When users resign, for example, you only need to remove them from one place. Similarly, when a new user joins the company or gets transferred to another department, you need to change the roles they’re associated with.

Let’s create a role that enables our user to execute the get pods command:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: get-pods
rules:
 - apiGroups: ["*"]
   resources: ["pods"]
   verbs: ["list"]

Let’s apply this definition using kubectl command:

kubectl apply -f role.yaml

Now, we have a role that enables its users to list the pods on the default namespace. But, in order for the magalix user to be able to execute the get pods, it needs to get bound to this role.

Kubernetes offers the RoleBinding resource to link roles with their objects (for example, users). Let’s modify the role.yaml file to look as follows:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: get-pods
rules:
 - apiGroups: ["*"]
   resources: ["pods"]
   verbs: ["list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: magalix-get-pods
subjects:
- apiGroup: ""
 kind: User
 name: magalix
roleRef:
 apiGroup: ""
 kind: Role
 name: get-pods

Although it’s perfectly possible to have the RoleBinding resource in a different file than the Role, it is considered a good Kubernetes practice to group similar or interdependent resources in the same file for easier management. Let’s apply this definition now:

kubectl apply -f role.yaml
role.rbac.authorization.k8s.io/get-pods created
rolebinding.rbac.authorization.k8s.io/magalix-get-pods created

Now, let’s see if magalix is able to list the pods on the cluster:

kubectl --user=magalix get pods
NAME          READY   STATUS             RESTARTS   AGE
hostpath-pd   1/1     Running            0          2d
mysqlclient   0/1     CrashLoopBackOff   75         6h

By looking at the above output, we can see that we have a pod, mysqlclient that apparently have issues. Assuming that we no longer need this pod, let’s delete it:

$ kubectl --user=magalix delete pods mysqlclient
Error from server (Forbidden): pods "mysqlclient" is forbidden: 
User "magalix" cannot delete pods in the namespace "default"

As you can see, the user is not able to delete the pods, yet it was able to list them. To understand why this behavior happened, let’s have a look at the get-pods Role rules:

  • The apiGroups is an array that contains the different API namespaces that this rule applies to. For example, a Pod definition uses apiVersion: v1. In our case, we chose [*], which means any API namespace.
  • The resources is an array that defines which resources this rule applies to. For example, we could give this user access to pods, jobs, and deployments.
  • The verbs in an array that contains the allowed verbs. The verb in Kubernetes defines the type of action you need to apply to the resource. For example, the list verb is used against collections while "get" is used against a single resource. So, given the current access level granted to magalix, a command like kubectl --user=magalix get pods hostpath-pd will fail while kubectl --user=magalix get pods will get accepted. The reason is that the first command used the get verb because it requested information about a single pod. For more information about the different verbs used by Kubernetes check the official documentation.

Let’s assume that we need magalix to have read-only access to the pods, both as a collection and as a single resource (get and list verbs). But we don’t want it to delete Pods directly. Instead, we grant it access to the Deployment resource and, through Deployments, it can delete and recreate pods (like though rolling updates). A policy to achieve this may look as follows:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: get-pods
rules:
 - apiGroups: ["*"]
   resources: ["pods"]
   verbs: ["list","get","watch"]
 - apiGroups: ["extensions","apps"]
   resources: ["deployments"]
   verbs: ["get","list","watch","create","update","patch","delete"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: magalix-get-pods
subjects:
- apiGroup: ""
 kind: User
 name: magalix
roleRef:
 apiGroup: ""
 kind: Role
 name: get-pods

We made two changes here:

  • Added the get and watch to the allowed verbs against Pods.
  • Created a new rule that targets Deployments and specified the necessary verbs to give the user full permissions.

Now, let’s test the different actions that our user is allowed or not allowed to do:

First, we create a simple Nginx deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
 labels:
   app: nginx
spec:
 replicas: 3
 selector:
   matchLabels:
     app: nginx
 template:
   metadata:
     labels:
       app: nginx
   spec:
     containers:
     - name: nginx
       image: nginx:1.7.9
       ports:
       - containerPort: 80

The deployment creates three Pods each hosting one container running Nginx web server.

$ kubectl --user=magalix  apply -f deployment.yaml
deployment.apps/nginx-deployment created

Once the deployment is created, let’s list the Pods:

$ kubectl --user=magalix  get pods
NAME                                READY   STATUS             RESTARTS   AGE
hostpath-pd                         1/1     Running            0          2d
mysqlclient                         0/1     CrashLoopBackOff   85         6h
nginx-deployment-75675f5897-hh6d5   1/1     Running            0          7s
nginx-deployment-75675f5897-mh8xl   1/1     Running            0          7s
nginx-deployment-75675f5897-skqjm   1/1     Running            0          7s

Get a single Pod:

kubectl get pods nginx-deployment-75675f5897-hh6d5
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-75675f5897-hh6d5   1/1     Running   0          34s

What about deleting this pod?

$ kubectl --user=magalix  delete pods nginx-deployment-75675f5897-hh6d5
Error from server (Forbidden): pods "nginx-deployment-75675f5897-hh6d5" is forbidden: 
User "magalix" cannot delete pods in the namespace "default"

OK so we cannot directly delete Pods. But we should be able to replace them by modifying the Deployment definition to look like the following:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
 labels:
   app: apache
spec:
 replicas: 3
 selector:
   matchLabels:
     app: apache
 template:
   metadata:
     labels:
       app: apache
   spec:
     containers:
     - name: apache
       image: httpd
       ports:
       - containerPort: 80

The change we made was replacing the nginx image with httpd (Apache), and modifying the Pod labels and the Pod selector accordingly. This change effectively deletes the existing Pods that use nginx and replace them with new ones hosting containers running Apache:

$ kubectl --user=magalix  apply -f deployment.yaml
deployment.apps/nginx-deployment configured
$ kubectl --user=magalix get pods
NAME                                READY   STATUS             RESTARTS   AGE
hostpath-pd                         1/1     Running            0          2d
mysqlclient                         0/1     CrashLoopBackOff   85         6h
nginx-deployment-75675f5897-mh8xl   1/1     Terminating        0          3m
nginx-deployment-75675f5897-skqjm   1/1     Running            0          3m
nginx-deployment-c4f6cdd64-8w92k    1/1     Running            0          6s
nginx-deployment-c4f6cdd64-brnmd    0/1     Pending            0          1s
nginx-deployment-c4f6cdd64-dj9wr    1/1     Running            0          3s

As you can see, the new deployment definition is actively replacing Pods.

Notice that in all the preceding examples, we didn’t specify a namespace, so our Role is applied to the default namespace. A Role is bound to the namespace defined in its configuration. So, if we changed the metadata of our Role to look like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: get-pods
  namespace: web

, the magalix user wouldn’t have access to the pods or the deployments unless working in the web namespace.

But, sometimes you need to specify Roles that are not bound to a specific namespace but rather to the cluster as a whole. That’s when the ClusterRole comes into play.

Cluster-Wide Authorization Using ClusterRoles

ClusterRoles work the same as Roles, but they are applied to the cluster as a whole. They are typically used with service accounts (accounts used and managed internally by the cluster). For example, the Kubernetes External DNS Incubator project uses a ClusterRole to gain the necessary permissions it needs to work. The External DNS Incubator can be used to utilize external DNS servers for Kubernetes service discovery. The application needs read-only access to Services and Ingresses on all namespaces, but it shouldn't be granted any further privileges (like modifying or deleting resources). The ClusterRole for such an account should look as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: external-dns
rules:
- apiGroups: [""]
 resources: ["services"]
 verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
 resources: ["ingresses"]
 verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: external-dns-viewer
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: external-dns
subjects:
- kind: ServiceAccount
 name: external-dns

The above definition contains three definitions:

  • A service account to use with the container running the application.
  • A ClusterRole that grants the read-only verbs to the Service and Ingress resources.
  • A ClusterRoleBinding works the same as a RoleBinding but with ClusterRoles. The subject here is ServiceAccount rather than User, and its name is external-dns.

Another everyday use case with ClusterRoles is granting cluster administrators different privileges depending on their roles. For example, a junior cluster operator should have read-only access to resources to get acquainted; then more access can be granted later on.

TL;DR

  • Kubernetes uses RBAC to control different access levels to its resources depending on the rules set in Roles or ClusterRoles.
  • Roles and ClusterRoles use API namespaces, verbs, and resources to secure access.
  • Roles and ClusterRoles are ineffective unless they are linked to a subject (User, serviceAccount...etc) through RoleBinding or ClusterRoleBinding.
  • Roles work within the constraints of a namespace. It would default to the “default” namespace if none was specified.
  • ClusterRoles are not bound to a specific namespace as they apply to the cluster as a whole.

Related posts

KubeCon EU 2023 Recap – GitOps Sessions on Flux with OCI, Liquid Metal CI/CD Platforms & Telco Cloud Platforms

Extending GitOps Beyond Kubernetes with Terraform Controller

Kubernetes On-Premise - What You Need to Know

Whitepaper: Production Ready Checklists for Kubernetes

Download these comprehensive checklists to help determine your internal readiness and gain an understanding of the areas you should update

Download your Copy