Introduction to Kubernetes Security
Weaveworks and Aqua recently held a popular webinar on Kubernetes and Security. Liz Rice of Aqua described how to lock down a cluster and Brice Fernandes of Weaveworks examined how to secure your operations workflows to Kubernetes.
Weaveworks and Aqua last week held a very well attended webinar on Kubernetes and Security. Speakers included Liz Rice (@lizrice) of Aqua Security and Brice Fernandes (@fractallambda) of Weaveworks.
Kubernetes is fundamentally a complex system with lots of different potential attack vectors aimed at data theft currency mining and other threats. Brice Fernandes started us off with a discussion on how to secure your operations to Kubernetes using GitOps best practices. Liz Rice then followed up on the current state of Kubernetes security related features as well as best practices and other tips on how to secure your cluster.
GitOps is an Operations Model for Kubernetes
According to Brice, Kubernetes clusters were traditionally accessed by developers directly, using the command line tool `kubectl`. There are of course many issues with having your development team directly accessing the cluster in this way. The biggest problem with this is that it is really hard to audit and track who did what, when.
GitOps eliminates this direct insecure access and instead provides a way for developers to safely access the cluster through a Git repository.
GitOps can be summarized as these two things:
- An operating model for Kubernetes and other cloud native technologies, providing a set of best practices that unify deployment, management and monitoring for containerized clusters and applications.
- A path towards a developer experience for managing applications; where end-to-end Git workflows are applied to both operations, and development.
The development team makes changes to applications through pull requests in the Git repository. Software deployment agents kept inside of the cluster are then responsible for applying those changes safely to the cluster. With deployment agents kept inside of the cluster, developers no longer need direct access. This also prevents credentials from being easily exploited through your CI system, where cluster secrets may be exposed as environment variables in custom scripts outside of the cluster’s trust domain.
What you need for GitOps
For GitOps workflows, these are the things you need:
- Your entire system described declaratively.
- The desired system state versioned-controlled in Git.
- Any changes to the desired state applied as version-controlled commits.
- Software agents that alert on divergence and synchronize the desired state in Git with a running cluster.
For more information, see, GitOps - What you need to know
Why GitOps is a more secure pipeline
As mentioned, CICD with GitOps addresses insecure pipelines by running a deployment agent within the cluster. It operates on a configuration Git repo, with its own credentials. The software agents compare and reconcile the desired state as expressed in the manifest files kept in Git against the actual state of the cluster.
In this way, cluster credentials don’t leak across the boundaries within your deployment pipeline. For example, your CI system can operate in a different security ‘zone’ rather than on the target cluster. Also, each pipeline component will only need a single RW credential. In addition, Git also provides you with exceptional auditing capabilities of who did what, and when and other security guarantees.
GitOps and securing your Git repository
As you can see, GitOps is a more secure way to update deployments to Kubernetes, but it does mean that the burden of security has shifted on to Git itself.
The main areas in Git that you need to pay attention are:
- User impersonation
- Malicious user tampering with the repository’s history
- Malicious user attacks on the Git platform
- Historical attacks on Git clients
Introduction to Kubernetes Security
Next, Liz Rice explained all of the major attack surfaces for Kubernetes that you need to consider before running your cluster in production. Because Kubernetes is a complex system with many APIs, there are several different attack surfaces from which your deployments can be compromised.
The main areas you need to focus on is your Kubernetes configuration setup and your application images. Both of these need to be set up properly to ensure that these are locked down sufficiently so that attackers don’t find any weak spots.
Tesla Cloud hacked by cryptocurrency mining software
Several months ago there was a famous attack on Tesla cloud where an attacker had installed cryptocurrency mining malware on their servers. The attackers got in because of exposed cluster admin credentials on the Kubernetes dashboard that was left open to the Internet after a fresh installation of Kubernetes.
But Liz explained that a lot of improvements have been made to out-of-the box security for Kubernetes. For example, kubeadm, a popular installer no longer installs the kube dashboard by default. It also now makes many of the default settings more robust so they can’t be hacked.
However, Liz stressed that one of the most important things you can do is keep your Kubernetes version up to date.
What do you need to secure?
Although most settings are now more robust after you’ve installed Kubernetes, you still need to lock down the Control Plane and you also need to ensure your application containers are secure.
Control Plane
- API server
- Etcd APIs
- Kubelet APIs
- Implement RBAC
Container images
- Limit your set of base images
- Define the USER in containers and don’t run as root
- Automate and use a vulnerability scanner
- It’s good practice to use a private image registry
- Pin dependencies for reproducible builds
- Define network policy for Pods and containers
- Manage secrets carefully
To research what settings need attention in your control plane, Liz suggests that you take a look at the guide put out by the Centre of Internet Security, CIS Benchmarks. You can also run the open source tool, kube-bench to see how compliant and secure your cluster is.
For more best practices on securing your cluster for deployments, view the entire webinar:
