Integrate OPA Into Your Kubernetes Cluster Using Kube-mgmt
We'll cover how to deploy OPA from scratch, and apply a sample policy that enforces using an Ingress hostname from a whitelist.
Earlier in this series, we explained what OPA is, then we demonstrated how easy it is to integrate OPA with your Kubernetes cluster through the OPA Gatekeeper project. In this article, we explore another means of OPA-Kubernetes integration, but this time without using OPA Gateway. Despite being lengthy, this procedure will give you more control over the process and will also teach you the inner workings of how the integration is done. In this article, we’ll cover how to deploy OPA from scratch, and apply a sample policy that enforces using an Ingress hostname from a whitelist. For this lab, we’re using Minikube.
Part 1: Install OPA
Step 1: Ensure That You Have The Prerequisites
- Kubernetes version 1.13 or higher.
- The ValidatingAdmissionWebhook admission controller must be enabled. You can enable it, and other recommended admission controllers, when the API starts by following this guide.
- Since we’re using minikube, we’ll need to ensure that the ingress addon is enabled: minikube addons enable ingress
- OPA expects to load policies from ConfigMaps in the opa namespace. Let’s create this namespace now: kubectl create namespace opa
- Change the context to the opa namespace kubectl config set-context
Step 2: Create The Necessary TLS Certificate
To secure the communication between the API server and OPA, we’ll need to configure TLS:
- Create a certificate authority and key:
-
openssl genrsa -out ca.key 2048 -
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
-
- Generate the key and certificate for OPA:
cat >server.conf <<EOF [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOF$ openssl genrsa -out server.key 2048 $ openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf $ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf -
Create a Kubernetes TLS Secret to store our OPA credentials:
kubectl create secret tls opa-server --cert=server.crt --key=server.key
Step 3: Deploy The Admission Controller
The adminssion-controller-yaml file should look as follows:
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: opa-viewer
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: configmap-modifier
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: opa-configmap-modifier
roleRef:
kind: Role
name: configmap-modifier
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
name: opa
namespace: opa
spec:
selector:
app: opa
ports:
- name: https
protocol: TCP
port: 443
targetPort: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: opa
namespace: opa
name: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
# WARNING: OPA is NOT running with an authorization policy configured. This
# means that clients can read and write policies in OPA. If you are
# deploying OPA in an insecure environment, be sure to configure
# authentication and authorization on the daemon. See the Security page for
# details: https://www.openpolicyagent.org/docs/security.html.
- name: opa
image: openpolicyagent/opa:latest
args:
- "run"
- "--server"
- "--tls-cert-file=/certs/tls.crt"
- "--tls-private-key-file=/certs/tls.key"
- "--addr=0.0.0.0:443"
- "--addr=http://127.0.0.1:8181"
- "--log-format=json-pretty"
- "--set=decision_logs.console=true"
volumeMounts:
- readOnly: true
mountPath: /certs
name: opa-server
readinessProbe:
httpGet:
path: /health?plugins&bundle
scheme: HTTPS
port: 443
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
scheme: HTTPS
port: 443
initialDelaySeconds: 3
periodSeconds: 5
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:0.8
args:
- "--replicate-cluster=v1/namespaces"
- "--replicate=extensions/v1beta1/ingresses"
volumes:
- name: opa-server
secret:
secretName: opa-server
---
kind: ConfigMap
apiVersion: v1
metadata:
name: opa-default-system-main
namespace: opa
data:
main: |
package system
import data.kubernetes.admission
main = {
"apiVersion": "admission.k8s.io/v1beta1",
"kind": "AdmissionReview",
"response": response,
}
default response = {"allowed": true}
response = {
"allowed": false,
"status": {
"reason": reason,
},
} {
reason = concat(", ", admission.deny)
reason != ""
}
The file creates the necessary RBAC components, a Deployment, a ConfigMap, and a Service. There are two points that we need to emphasize in this definition file:
- The Service name (opa) must match the CN (or the common name) that we chose for the certificate. Otherwise, TLS communication will fail.
- In the Deployment, we have the kube-mgmt sidecar container loaded with the following command-line arguments:
- --replicate-cluster=v1/namespaces.
- --replicate=extensions/v1beta1/ingresses.
Now, apply the definition:
kubectl apply -f admission-controller.yamlStep 4: Deploy The Admission Webhook
For the admission controller to work, we need an admission webhook that receives the admission HTTP callbacks and executes them. Now, let’s create our webhook configuration file:
cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1beta1
metadata:
name: opa-validating-webhook
webhooks:
- name: validating-webhook.openpolicyagent.org
namespaceSelector:
matchExpressions:
- key: openpolicyagent.org/webhook
operator: NotIn
values:
- ignore
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
clientConfig:
caBundle: $(cat ca.crt | base64 | tr -d '\n')
service:
namespace: opa
name: opa
EOF
This webhook configuration has the following properties:
- It won’t listen to actions coming from any namespace that has the openpolicyagent.org/webhook=ignore label. That’s necessary so that OPA doesn’t intercept requests in the kube-system namespace, or its own namespace (will add labels to them later).
- It will listen for the CREATE and UPDATE actions on all resources.
- It uses a base64 representation of the CA certificate, that we created earlier, to be able to communicate with OPA.
Now, before we apply the configuration, let’s label the kube-system and opa namespaces so that they’re not within the webhook scope:
kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore
And apply the following configuration to register OPA as an admission controller.
kubectl apply -f webhook-configuration.yaml
Part 2: Deploy A Sample Policy
Step 1: Define The Policy In Rego
OPA uses the Rego language to describe policies. For our lab, we’ll use the example mentioned in the official documentation. Our ingress-whitelist.rego should look as follows:
package kubernetes.admission
operations = {"CREATE", "UPDATE"}
deny[msg] {
input.request.kind.kind == "Ingress"
operations[input.request.operation]
host := input.request.object.spec.rules[_].host
not fqdn_matches_any(host, valid_ingress_hosts)
msg := sprintf("invalid ingress host %q", [host])
}
valid_ingress_hosts = {host |
whitelist :=input.request.namespace.metadata.annotations["ingress-whitelist"]
hosts := split(whitelist, ",")
host := hosts[_]
}
fqdn_matches_any(str, patterns) {
fqdn_matches(str, patterns[_])
}
fqdn_matches(str, pattern) {
pattern_parts := split(pattern, ".")
pattern_parts[0] == "*"
str_parts := split(str, ".")
n_pattern_parts := count(pattern_parts)
n_str_parts := count(str_parts)
suffix := trim(pattern, "*.")
endswith(str, suffix)
}
fqdn_matches(str, pattern) {
not contains(pattern, "*")
str == pattern
}
If you’re new to Rego, the code may seem cryptic at first, but Rego makes it really easy to define policies. Let’s spend a few moments highlighting how this policy enforces using Ingress namespace from the whitelist:
- Line 1: The package is used the same way that it’s used in other languages.
- Line 3: We define a data set that contains two items: CREATE and UPDATE
- Line 5: This is the policy itself - it starts with deny followed by the policy body. If the combination of the statements in the body evaluates to true, the policy is violated, the action is blocked, and the message is returned to the user with the reason why the action was blocked.
- Line 6: The input object is special. Any JSON message sent to OPA starts with the input object at the root. We’re traversing the JSON object until we reach the resource in question and it must be “Ingress” for the policy to be applied.
- Line 7: We need to apply the policy to create or update the resource. In Rego, we can do that through the shorthand operations[input.requset.operation] where the code inside the square brackets extracts the operation specified in the request. The statement is true if it matches one of the items defined in the operations set in line number 3.
- Line 8: To extract the host(s) that the Ingress object will have, we iterate through the rules array of the JSON object. Again, Rego provides the _ character to loop through the array and return all the items to the host variable.
- Line 9: Now that we have our host, we need to ensure that it’s not one of the whitelisted hosts. Remember, the policy is violated only if it evaluates to true. In order to check the existence of the host in the valid-hosts list, we use the fqdn_matches_any function that is defined in line 19.
- Line 10: Defines the message that should be returned to the user explaining why the Ingress object could not be created.
- Lines 13-17: This part extracts the whitelisted hostnames from the annotations part of the Ingress namespace. The hostnames are added in a comma-separated list and the split built-in function is used to convert the hosts to a list. Finally, the _ is used to iterate through all the extracted hosts. The result is piped to the host variable through the | character. If you’ve programmed in Python before, this is very similar to list comprehensions.
- Line 19: The function simply accepts a string and searches for it in a list of patterns, which is the second argument. To do that, it uses another function, fqdn_matches defined in line 23 and also in line 33. In Rego, you can define multiple functions with the same signature as long as all of them produce the same output. When you call a function that’s defined more than once, all the occurrences of the function are called. More on this in the official docs.
- Lines 23-31: The first fqdn_matches definition.
- First, it extracts the hostname from the pattern by splitting the pattern to tokens by the dot (.) so *.example.com becomes *, example, and com.
- Next, it ensures that the first token of the pattern is an asterisk. The splitting operation is done on the str part (which would be the fqdn name when the function is called).
- It counts the number of tokens in the pattern and the input string.
- If extracts the suffix from the pattern by removing the *. Part.
- Finally, it evaluates whether the input string input ends in the suffix or not. So, if the allowed pattern is *.mydomain.com and the string is www.example.com, the policy is violated since www is not part of mydomain.com.
- Lines 33-36: The second validation function. This function is used to validate patterns that do not use wildcards. For example, when the pattern is written as mycompany.mydomain.com
- First, we need to ensure that the supplied pattern does not contain a wildcard. Otherwise, the statement will evaluate to false and the function will not continue.
- It the pattern refers to a specific domain, then we just need to ensure that the fqdn matches this pattern. In other words, if the pattern is mycompany.mydomain.com then the fqdn of the host must also be mycompany.mydomain.com.
The reason we have two functions with the same name and signature is because of a limitation in the Rego language that prevents functions from producing more than one output value. So, to make more than one validation at the same time with different logic, you must use multiple functions with the same name.
Step 2: Apply The Policy
In a real-world case, you should thoroughly test your Rego code before applying it to the cluster. There are several ways to perform quality assurance on Rego code including Unit Testing. You can (and should) also use the Rego Playground to try the code and spot any errors using sample data.
To apply the policy to the cluster, you need to create a ConfigMap with the file contents in the opa namespace:
kubectl create configmap ingress-whitelist --from-file=ingress-whitelist.rego
Step 3: Ensure That The Policy Is Working As Expected
Next, let’s create two namespaces: one for the QA environment and the other for production. Notice that both of them contain the ingress-whitelist annotation, holding a list of the domain patterns that the Ingress hostname should strictly be part of.
1. qa-namespace.yaml:
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-whitelist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
name: qa
2. production-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-whitelist: "*.acmecorp.com"
name: production
Apply both files to the cluster:
kubectl apply -f qa-namespace.yaml -f production-namespace.yaml
Next, let’s create an Ingress that uses one of the allowed domains:
kubectl apply -f - <<EOT
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-ok
namespace: production
spec:
rules:
- host: signin.acmecorp.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
EOT
Ensure that the Ingress was created:
kubectl get ing -n production
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-ok <none> signin.acmecorp.com 192.168.99.101 80 54s
Now, let’s try to create an Ingress that should not be allowed:
$ kubectl apply -f - <<EOT
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-bad
namespace: qa
spec:
rules:
- host: acmecorp.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
EOT
Error from server (invalid ingress host "acmecorp.com"): error when creating "STDIN":
admission webhook "validating-webhook.openpolicyagent.org" denied the request:
invalid ingress host "acmecorp.com"
As you can see from the output, the API server refused to create the Ingress object since the hostname we tried to use for the QA Ingress is reserved for production.
Troubleshooting
Your policy is not working as expected, or at all? Most likely you have some kind of error in your Rego code. You can always check the status of the policy by examining the properties of the ConfigMap that contains the policy:
kubectl get cm ingress-whitelist -o yaml
TL;DR
- OPA is a general-purpose, platform-agnostic policy enforcement tool.
- Kubernetes is among the many technologies that OPA can integrate with.
- You can integrate OPA into Kubernetes using the OPA Gatekeeper project. This was covered in our previous article.
- You can also deploy OPA by manually creating the admission controller, the admission webhook, and a few other necessary components that we covered in this article.
- Once OPA is deployed, you can use it to enforce different types of policies that differ from what RBAC does. For example, you can deny a request that was authenticated and authorized but contains a property that you disallow.
- In this article, we explored a use case where you can use OPA to deny creating an Ingress object (although the user has the required privileges to create it) when it contains a hostname that you don’t allow.
- OPA is super-powerful - to fully utilize its potential, you need to understand the Rego language. The language reference is an excellent place to start and you should try and exercise different policies on your own.
- The Rego Playground is a very helpful learning and debugging tool for the Rego language.
Whitepaper: Trusted Delivery with GitOps and Policy as Code
Download our latest whitepaper and learn how automated security and compliance checks, in the form of policy as code, make automated continuous deployments safe and secure.
