Build a seamless GitOps pipeline with Flux
Get a good idea of what a GitOps pipeline looks like, how it differs from a CI/CD pipeline, and why Flux is central to a GitOps pipeline.
GitOps is a set of principles for you to implement continuous deployment for cloud-native applications using Git as a single source of truth. With GitOps, you treat everything, including infrastructure, as source code and store it in Git. GitOps lets you automatically deploy code as soon as it is committed to a Git repository. Here are some of its benefits:
- It offers a developer-centric approach
- It promises high reliability with faster error recovery
- It gives you complete visibility
- It greatly increases your productivity
When adopting GitOps in any organization, it takes setting up a GitOps pipeline that is different from a more traditional CI/CD pipeline. Modern cloud-native tools make up this new pipeline. In this post, we list the most vital tools that makeup a GitOps pipeline. We also discuss the practices that go along with these tools, which is sometimes more important than the tool itself. The list is not meant to be exhaustive but is a good indicator of the leading GitOps tools today.
1. Git Management
Online Git hosting platforms are vital to GitOps as they enable you to not just store your code but also make it easy for teams to collaborate around these cloud-hosted repositories. They manage changes to repositories, store a version history, and are built to deal with complex branching of repositories.
Here are the two most popular Git tools:
GitHub
GitHub needs no introduction as the leading Git tool in use today. Originally viewed as a place to store open source code, and external software components, GitHub has been making its way to internal software pipelines, and particularly in GitOps pipelines. It brings useful features like issue tracking, code review, and general code management. Git’s branching system revolutionized software development about a decade ago, and today that feature is still highly useful in structuring GitOps workflows. Other alternatives to GitHub include BitBucket, and GitLab.
Some may believe that if they’re using a tool like GitHub, they’re automatically following GitOps practices. However, this is far from the truth as the rest of this list shows. It takes a lot more than a Git tool to practice GitOps.
2. Container Registry
A container registry allows you to store all the information, files, and components of an application in the form of container images. Container images are used as templates to scale application development quickly. These container images must be stored in a secure and shareable registry.
Docker Hub
Docker Hub is a cloud-based registry service that is used to store, retrieve and share container images with your team. Upon integration, Flux can automatically scan and download container images from a registry, make changes to the image (eg. change version tags), and deploy the image to a production cluster.
Other popular container registries include the ones from the cloud vendors like AWS Elastic Container Registry (ECR), or Google Cloud Container Registry or even DIY.
3. A GitOps Agent
This is arguably the most important part of a GitOps pipeline. And the leading GitOps agent today is Flux.
Flux
Once code is committed to Git, a GitOps agent like Flux takes over to run automated steps and move the code down the supply chain. Flux is an incubating project at the Cloud Native Computing Foundation that ensures the state of a cluster matches the configuration in Git. It forms a complete Continuous Delivery platform that enables application deployment. It works with all Git providers and various container registries and can be seamlessly integrated with other automation tools as well.
4. Kubernetes Configuration Automation
One of the core principles of GitOps is the automated application of desired changes across the pipeline to match the single source of truth described in Git. To ensure effective automation Flux uses Kubernetes automation solutions.
Helm
Helm helps you manage Kubernetes applications using a packaging format called charts. A chart comprises descriptive files of pre-configured Kubernetes resources. These charts are stored in Helm Repository, and from here you can share the resources details of the applications with your team.
Helm is built with a pluggable architecture, allowing a declarative management tool like Flux to use its Helm Controller and take advantage of the Helm SDK. As a package management solution, Helm streamlines Kubernetes application deployment.
Kustomize
Flux doesn’t stop with Helm, but also leverages Kustomize to further automate and customize configuration changes for Kubernetes. Kustomize can be defined in simple terms as a pipeline for infrastructure. Kustomize keeps a base layer of a resource like a Helm Chart, which remains unchanged, and then adds multiple patch layers that customize the base layer. The advantage of this is that when Kustomize needs to be updated, only the base layer is updated, and the customizations in the patch layers are not lost and need not be reconfigured. This is a way to manage Kubernetes configuration at scale.
Kustomize leverages plain YAML files, which means it is very simple to learn and use. Flux includes a Kustomize controller to manage continuous delivery pipelines for Kubernetes.
5. Progressive delivery
Progressive delivery uses a phased out deployment strategy to limit the impact of failures on end users.
Flagger
With GitOps, the leading progressive delivery tool is Flagger. It is easy to integrate Flagger with Flux and manage really complex deployments. Flagger enables you to automate canary deployments, measure performance metrics, and implement flexible traffic routing. It primarily ensures that any change made in Git is deployed selectively in a series of steps. Flagger integrates with service meshes like Linkerd, Istio, App Mesh, and Contour to manage traffic shifting during canary deployments, A/B testing and blue-green deployments.
Importantly, Flagger also supports rollback of bad changes that can break a system. Sometimes this rollback happens so fast that Admins only find out after the rollback that there was a very brief failure. This is the kind of ideal scenario a well-setup GitOps pipeline can enable.
6. Monitoring tools
Once a GitOps pipeline is set up, and deployments are happening, it is vital to monitor every step of the pipeline for bugs, errors, latency, and security risks. This requires robust monitoring tools.
Prometheus & Grafana
Prometheus is the leading time-series monitoring tool, and Grafana is the leading visualization tool in the cloud-native space today. Flux leverages these powerful solutions to enable state-of-the-art monitoring for GitOps.
Prometheus is able to pull real-time performance metrics from all Flux controllers such as Helm Controller, and Kubernetes API. These metrics are queried by Grafana where they can be visualized in the form of dashboards.
Source: Flux
This dashboard, for example, shows the number of cluster reconcilers, failed reconcilers, Kubernetes manifest sources, and more. Grafana can also generate alerts if a reconciliation fails. These metrics are essential to assess the status of deployments, and make decisions in real-time.
We hope this post gives you a good idea of what a GitOps pipeline looks like. As you probably noticed, a GitOps pipeline is quite different from a more traditional CI/CD pipeline. Both pipelines have the same goal of moving code from Dev to Production in the fastest way possible, but they take different approaches. A GitOps pipeline is very Kubernetes-centric, leveraging various open source tools like Helm, Kustomize, and Flagger.
Another point to note is that Flux is the most important piece of a GitOps pipeline. It integrates with every other tool and pretty much brings together the entire GitOps pipeline. It automates various activities that are usually done manually, and delivers great control to tweak and customize to any extent.
If you would like to learn more about Flux in a hands-on workshop, join the Weave Online User Group events.
Finally, if you’re looking for a readymade GitOps pipeline that allows you to completely skip the integration and setup phase, look no further than Weave GitOps. It comes with all the tools mentioned in this post by default, or makes them available as plugins. There are two flavors - Weave GitOps Core (here is even a quick getting started guide), which is the open source option and an opinionated wrapper around Flux, and Weave GitOps Enterprise, which comes with all the bells and whistles such as team workspaces, RBAC, and more.
If you’re looking to adopt GitOps, a GitOps pipeline is indispensable. Now you know which tools make up a GitOps pipeline, and how to get started the easy way.
