CI/CD for Kubernetes

Learn what you need to know about continuous deployments and what tools we recommend for automating a delivery pipeline to Kubernetes.

Automation and Deployments for Kubernetes

Who we are?

As the creators of GitOps, we help teams adopt and manage cloud-native infrastructure and applications quickly, securely, reliably, and at scale. Through our GitOps solutions, we enable infrastructure, and application development  teams to build and operate their own Kubernetes application platform whether in the cloud, at the edge, or on-premise. 

Our products and solutions are community-built and enterprise-approved. Weaveworks was one of the first members of the Cloud  Native Computing Foundation and is one of its top 10 contributors. Weaveworks is also a key partner with Microsoft Azure providing support for Flux in Microsoft GitOps deployments.

Why Automate your CI/CD Pipeline?

Automating Continuous Integration and Continuous Delivery (CI/CD) in a development environment is the ultimate goal for DevOps teams. Automation allows new features to be tested, deployed and delivered to your customers much more quickly. The ability to deliver more, faster, not only gives you a competitive advantage but it also leaves more time for innovation within your organization; and since adjustments and updates are rolled out rapidly, your customers will also be happier.

On this page we’ll take you through:

1. Manually updating your app in Kubernetes and why that’s a painful process.
2. Achieving Continuous Integration/Continuous Deployment (CI/CD) - the components and features you need.
3. Approaches to continuous delivery and an overview of Kubernetes CI/CD tools available to build your own automated pipeline.

1. Manual Kubernetes Deployments can be Error Prone

Kubernetes allows you to update your applications in real-time, otherwise known as rolling updates, but doing so manually is not ideal for a number of reasons:

• Almost everything related to deployments or most user facing components in Kubernetes are defined in YAML files. There are potentially multiple YAMLs per Kubernetes application and if you are not used to white-space syntax, they can be frustratingly error prone with no feedback when things go wrong.
• Kubernetes applications are comprised of several different resource types, some of which may need upgrading, and depending on the type of app you are working on, knowing what to update can make Kubernetes deployments  particularly complex.
• Keeping track of the diff history, managing rollbacks of new and old deployments especially when you are working on a large application within a team can be challenging.
• Managing secrets in Kubernetes between team members working on the same app across different repos requires that you are careful not to leak credentials.

An advantage with Kubernetes deployments though is that they are declarative, which means they can be kept under source control. Not only is it a best practice to store declarative configuration in source control, but when disaster strikes and your app is wiped out, keeping configuration in Git makes it easier to recover using the tools that developers know best.

Links

• GitOps: Operations by Pull Request
• Kubernetes Rolling Upgrades One by One

---

Learn how to automate Kubernetes infrastructure and deployments in this whitepaper, "Automating Kubernetes with GitOps". 

---

2. Achieving Continuous Delivery in Kubernetes

Standing up a Kubernetes cluster today is a rather straightforward operation, but what isn’t so simple is getting your updates or new features rolled out to your running cluster. There are a number of options available that can help you automate a continuous delivery pipeline to Kubernetes without you and your team having to manually deploying changes.

These are the components that make up a Continuous Delivery (CD) pipeline:

• Version Control System – a source code repository where changes and updates are pushed
• CI system – an integration and test system that may also build the Docker image
• Docker Registry – the image registry that stores your Docker images
• Kubernetes Cluster – set up with a few clicks in GKE or using one of the available installers

The key to Continuous Delivery is to coordinate and automate these pieces to increase velocity and make Kubernetes deployments seamless.   

Most developers use Git for version control, and you may also be using an automated testing and integration service like Jenkins, Travis or CircleCI—any of these CI systems are capable of running through a set of tests and even building a Docker image.  Once tested and sanitized, the newly built image is sent to a container repository: either a public one like Google Container Registry (GCR), or to Quay or Docker Hub; or you may have an on-premises private registry.

At Weaveworks, we’re advocates of the GitOps methodology that puts Git as the catalyst for the pipeline. Git is the source of truth for declarative configuration for Kubernetes, deployments manifests as well as source code. When a developer makes a code change and pushes it to ‘git’ or any other Version Control System, it triggers the deployment pipeline.

We’ll discuss this in more detail later on this page.

Links:

• GitOps: Operations by Pull Request

Blue-Green Deployments and Canary Testing

In addition to the basic components, you may also need to support additional features like blue-green deployments and canary testing.

A blue-green deployment is when you run two identical deployments in parallel: one on production and another on a staging server. You can then carry out final testing on one server and afterwards switch the live server over to the tested version. If you need to rollback at a later point, for some reason, this deployment strategy allows you to do that quickly and without much downtime. 

(from Blue Green Deployments by Martin Fowler)

Canary deployments are used when you want to test out a new feature with a subset of users. Similar to a blue-green deployment, you might have two almost identical servers: one that goes to all users and another with the new features that gets rolled out to only a set of users. These types of deployments allow you to test new features with a trusted sub-set users to ensure that you’re building the right product.

 

(from canary (canary test, canary deployment))

Links:

• Blue-Green Deployment by Martin Fowler

3. Approaches to Continuous Delivery to Kubernetes

Even though there are a number of tools available, CI/CD vendors are still in the process of defining how a CI/CD Kubernetes pipeline can and should look like for engineers who create cloud native applications. At Weaveworks, we see two different paradigms emerging; one is a push type pipeline and the other is a pull pattern.

Push Pipelines

Most of the CI/CD tools available today use a push-based model. What we mean by push-based is that code goes through the pipeline starting with the CI system and may continue its path through a series of encoded scripts or by using ‘kubectl’ by hand to push changes to the Kubernetes cluster.

The reason you don’t want to use your CI system as the deployment impetus or do it manually on the command line is because of the potential to expose credentials outside of your cluster. While it is possible to make those interactions secure in both your CI/CD scripts and manually on the command line, you are working outside the trust domain of your cluster. This is generally not good practice and is why CI systems can be known as attack vectors for production.

 

Weave GitOps Pull Pipeline

Weave GitOps, on the other hand, uses a pull strategy and it consists of two key components: a Deploy Automator that watches the image registry and a Deploy Synchronizer that sits in the cluster to maintain its state.

At the centre of our pull pipeline pattern is a single source of truth for manifests (or a config repo). Developers push their updated code to the code base repository; where the change is picked up by the CI tool and ultimately builds a Docker image. The Weave GitOps ‘Deploy Automator’ notices the image, pulls the new image from the repository and then updates its YAML in the config repo. The Deploy Synchronizer, then detects that the cluster is out of date, and it pulls the changed manifests from the config repo and deploys the new image to the cluster.

 

Weave GitOps uses the deployment operator pattern to listen for and orchestrate service deployments to your Kubernetes cluster.

This approach is described by William Denniss in our KubeCon presentation.  By using the operator, an agent acts on behalf of the cluster to listen to events relating to custom resource changes, so they can be applied consistently.  In other words, the operator is responsible for synchronizing what’s in Git with what’s running in the cluster.

The operator deployment method is implemented by Flux, a CNCF open source project.

The benefits of a pull-based pipeline:

• With Weave GitOps deployment policies can be set and changed (automatic or manual) in one place instead of keeping them scattered across your CI pipelines and embedded in custom scripts.
• A Pull-based pattern means that cluster configuration is always in-sync with what’s being kept in Git.
• And most importantly, since CI is decoupled from CD, it means that the CI tool doesn’t have elevated access to your cluster and that secure credentials always remain with the cluster and are not widely shared across the pipeline or teams.

Links:

• The GitOps Pipeline
• The Gitops FAQs
• GitOps: High velocity CICD for Kubernetes
• GitOps - Application Delivery Compliance and Secure CICD - Part 4
• How Secure is Your Pipeline?
• Continuous Intrusion: a pentest