Kubernetes Workload Promotion in GitOps CD Pipelines
Read about our new Weave Cloud feature that lets you promote workloads between clusters. Find out how this can help your team accelerate the delivery of features to Kubernetes.
Weave Cloud has introduced a new feature called, Promotions, which is a step that’s integral to Continuous Delivery pipelines. This enables users to promote workload(s) in a multi-stage pipeline.
What is a “Pipeline”?
As we move towards progressive delivery, pipelines help developers and operators create and visualize the flow of workloads from source code to running in production clusters. Configuring pipelines can be done via a GitOps workflow model.
A workload is changed in the git repo and after a PR, the initial stage of a pipeline is triggered which starts moving through various stages of a pipeline. It allows developers and operators to interact with it at any time, at any stage (single or multi environments) and understand what’s happening with this workload, even after its fully operational.
Here’s a typical pipeline:
GitOps Deployment Workflow: Image Repo (source)->Cluster (target)
Git (Code) Repo-->CI (test, create package)-->Push to Image Registry (Docker, Quay, Jfrog, Nexus)-->Update Git (Config) Repo-->Weave Cloud (auto detects)-->Pulls to Cluster-->Sync with Git (Config) Repo
Challenges:
- If you can’t observe and monitor your workloads, how do you know it’s operating well?
- If you can’t visualize your workload deployments, how do you know if you can trust the system? Or know if something is broken during complex deployment(s)?
What is Promotion?
Promotion allows developers and operators to manage single or multiple workloads and “promote” them from one cluster to another - either automatically or manually. The workloads can be be promoted as individual containers, groups of containers - single or multiple (batch) workloads in a single promotion. The promotion of workloads are one or more steps towards a GitOps CD pipeline. Controlled promotion of workloads enables users to take actions during various stages of the pipeline. An example: execute a set of tests before promoting from Stage to Production.
This first iteration of Promotions allows you to choose your source and target cluster and promote your workloads between them. You can promote between any two sources and targets. With these steps you can promote your workloads all the way to production in a controlled manner. For example:
- Image Repo to Cluster A (Dev)
- Cluster A to Cluster B (Stage)
- Cluster B to Cluster C (Prod)
Promotion in Modern Software Development
Promotion of code from one environment to another has been around for a while and is used by operators to control what goes into production. It’s time we brought promotion into the modern age of software development where development teams in collaboration with operators own the pipeline all the way to production. To get here, a level of trust needs to be established so that DevOps teams and Operators can enable their developers to own the pipelines for each workload.
Managing Promotion within Pipelines is Complex
A developer can easily deploy a single workload to their minikube or dev cluster via the CLI. However, if you are part of a development team that is working with microservices divided among many workloads, it’s difficult to know what’s going on with multiple workloads being deployed at various times to any cluster in any cloud. Visualizing the workflow in a pipeline gives both operators and developers the confidence to know and share workload deployments with each other. When one team member kicks off a deployment and executes a pipeline, others can see what’s happening with that workload in real-time.
Differences between CI and GitOps CD Pipelines
We described previously that CI driven pipelines are an anti-pattern that @errordeveloper termed CIOps. Problems with CI based pipelines = “CIOps”:
- Security and Access to Kubernetes API and Cluster
- In-cluster CIOps
- What if Deployment updates don’t go as planned? Rollback issues
- What if multiple CI jobs are deploying the same version?
- What if someone got the config wrong and the CI job happens to override another’s Deployment?
- The Kubernetes cluster has no knowledge of Source of Truth (SoT) for the apps
- What happens if you lost your production cluster? - it’s happened to us and we were able to recover easily:
GitOps CD Pipelines to the rescue
Weave Cloud implements GitOps by using a Kubernetes operator inside the cluster that detects when the source of truth is different from the current state. GitOps CD Pipelines solve many of the problems with CI based pipelines, but CD pipelines are also not without its challenges:
- What’s happening with my workload during the rollout process?
- Is it stuck? Why is it taking so long?
- What about when I am rolling out to multiple clusters in a sequence? - manual or auto promote
- How is this update affecting resource consumption in the cluster?
- How do I see workload behavior after it’s fully operational?
- How do I get automatically notified if things go wrong? - it’s inevitable.
- What about when I deploy multiple workloads, helm charts, serverless functions to multiple regions/clouds?
Promotion in GitOps CD Pipelines
The stages of a pipeline can be very simple or they can be complex. It can be just a few stages or many stages. Workloads may be deployed to a single cluster or to many clusters or multiple clouds either in parallel or in sequence.
How Promotion works in Weave Cloud
Promote your workloads from the Image Repo to a Cluster:
- Switch to your target cluster
- Open ‘Deploy'
- Select Source -> Image Repo is default as it is the final Source of truth (SoT). You can change this from the drop down to pick a source cluster
- Select the workloads to Promote
- Select 'Promote Selected...'
- This takes you to the Release page where you can watch your workloads being promoted in progress
Promoting workloads from one cluster to another
This is a two step process:
Step 1: Deploy from Image Repo to a Cluster (Dev or Stage)
- Switch to the first target cluster in your pipeline - e.g. Stage
- Then promote from image repo or Dev to Stage (see steps above)
Step 2: Promote from one cluster to another (Dev to Stage, or Stage to Prod)
- Switch to the next target cluster in your pipeline - e.g. Prod
- Then follow the same steps to Promote
- Rinse and repeat for as many clusters in your pipeline
Give it a try at cloud.weave.works
What’s next?
- Visualizing full pipelines including interacting with the pipeline at any stage as well as the promotion stages
- Showing the live status of what’s happening with the workloads so that team members and operators can see at a glance what’s happening with the workload anywhere in the pipeline
