Securing GitOps Pipelines
Earlier this month Weaveworks hosted a webinar on securing your GitOps pipelines. Speakers included Andrew Martin of ControlPlane as well as Weaveworks’ customer success engineer Brice Fernandes.
Earlier this month, Weaveworks hosted a webinar on securing your GitOps pipelines. Speakers included Andrew Martin (@sublimino) of ControlPlane as well as Weaveworks’ customer success engineer, Brice Fernandes (@fractallamda).
Brice gave us an overview of what GitOps is, and why it is a logical and more secure way for large development teams to update applications in Kubernetes.
What do most CICD pipelines look like?
This is what a typical CICD pipeline looks like in most organizations. From the left side, the developer pushes his code into the repo, and from there, the CI tool takes over and runs some tests, integrates the code, builds a container artefact, and adds it to a container registry where it eventually gets deployed to the cluster.
As you can see from the diagram, the typical single direction, push-based pipeline has some security flaws. One of the biggest issues is the read-write credentials that are exposed from your CI tool right into your cluster, leaving you open to malicious attacks.
How does the security improve with GitOps?
Briefly, GitOps is an operations model, derived from computer science and years of operations knowledge. We developed this methodology at Weaveworks to manage our own Kubernetes applications and infrastructure for the last several years.
GitOps is technology agnostic (despite its name). It is a set of principles on why rather than step by step instructions on how. We use Git for our repositories, but these principles could be equally applied to Bitbucket or to Perforce or any other version control system.
GitOps is ultimately a way for your team to deliver more features, faster.
Principles of GitOps
These are the four driving principles of GitOps:
1. The entire system is described declaratively.
2. The canonical desired system state is versioned somewhere.
3. Approved changes to the desired state are automatically applied to the system.
4. The use of software agents to ensure correctness and alert on divergence.
Once you’ve applied the principles, this is what your system looks like in the end:
On one side, developers push code to the canonical source of truth. Software agents constantly compare the source of truth with the running cluster. When a change is detected between the two, an alert is sent where an action can be taken to re-adjust the cluster with the canonical source of truth.
The GitOps Pipeline
When these principles have been put into place, you have a pipeline that is already more secure. The security guarantees in such a pipeline are stronger. Credentials between tools are read-only all along the pipeline with critical credentials and secrets kept inside of the cluster itself.
In addition to having better security guarantees, since your configuration manifests are kept in its own canonical repo, you can also implement and enforce process and other constraints that you may have within your organization. For example, you can create separate policies for cluster infrastructure changes and for application deployments.
GitOps provides you with a more secure way of updating your cluster, however it does shift the burden of security onto Git.
Secure GitOps in Production
Next, Andrew Martin from ControlPlane spoke on securing your Git repositories.
Everything kept as code
Andrew began his talk with a discussion on how clusters and other infrastructures are almost completely declarative that can be kept in a software repository. But he stressed how keeping your Git repositories secure is now very important with this growing paradigm.
Git of course is the de facto standard for version control, however, it doesn’t really secure anything. Rather what it does is guarantee the provenance of the code commit.
How Git works
Git uses SHA-1 hash codes as the underlying mechanism for guaranteeing code commits. But SHA-1 hash codes can be vulnerable to collision attacks where when two SHA-1 hashes committed at the same time, one could overwrite the other and therefore put commit guarantee into question. This is a brute force method of attack that requires an enormous amount of computing power to accomplish. (The problem has since been updated with a newer version of SHA-1.)
How Git mitigates this collision risk, though is by adding dates and other metadata with a pointer to the parent code of the commit. Git operates like a Merkel tree much like the way blockchain does these days. This means there is a chain of commits that can be cryptographically verified as coming from the same parent.
Automate everything
Human intervention should be eliminated where-ever possible when it comes to updating anything in production. He stated that accidental/human error makes up for system errors almost as much as UPS system failures.
An example cited was the recent AWS S3 outage because of an incorrectly applied Ansible script by someone updating servers.
The solution to better security is to move humans as far away from production deployments as possible. - Andrew Martin, ControlPlane
Securing software pipelines
One of the most important concepts behind continuous delivery and your ability to secure the code on it is verifying where the code came from and whether it is legitimate. The immutability of Docker containers is very convenient, and it's very simple to send someone a container with code in it that can be run anywhere, for example. But how do you know whether the code within that container is from a reputable source?
Git threats that need mitigation
Andrew lists four commons threats that you should consider and mitigate when using Git:
- Threat #1: Git users can impersonate each other.
- Threat #2: Malicious user rewrites history.
- Threat #3: Malicious user removes security features.
- Threat #4: Old Git client versions are insecure.
Check out the full webinar recording below:
