Guide To GitOps

Have you heard about GitOps and are curious to know what it’s all about?  Read on as we describe the principles and patterns of GitOps workflows and how you can implement them to run Kubernetes in production and at scale. We’ll also describe the differences between GitOps and infrastructure-as-code (IAC) configuration management tools and of course show you how to adopt GitOps best practices as part of your own development environment.

What is GitOps?

Pioneered in 2017, GitOps is a modern way to manage cloud-native systems that are powered by Kubernetes. It leverages a policy-as-code approach to define and manage every layer of the modern application stack - infrastructure, networking, application code, and the GitOps pipeline itself. 

Read: The history of GitOps

GitOps leverages Git as the single source of truth to define every part of a cloud-native system. Once declared in Git, a GitOps agent (Flux) automatically applies all code, configuration, and policies across dev, test, staging, and production environments. With GitOps, whenever there is any divergence between Git and what's running in a cluster, developers are alerted. Depending on the case, Kubernetes reconcilers automatically update or roll back the cluster. With Git at the center of the delivery pipelines, developers can use familiar tools to make pull requests to accelerate and simplify both application deployments and policies that govern software delivery end-to-end.

An operating model for building cloud native applications

GitOps can be summarized as these two things:

1. It is an operating model for Kubernetes and other cloud-native technologies, that provides a set of best practices that bring together Git deployment, management, and monitoring for containerized clusters and applications.
2. It offers a path towards a self-service developer experience for managing applications and automating CI/CD pipelines, unifying development and operations teams.

The GitOps Working Group and the OpenGitOps Project

Having started in 2017, GitOps is still a relatively new approach to software delivery. As such, the principles that govern GitOps were much debated. Seeking to bring clarity to the core ideas, the CNCF has taken steps to set up a governing body to oversee the progress of GitOps in a vendor-neutral manner. To that end, they set up The GitOps Working Group, and the OpenGitOps project.

The GitOps Working Group is established under the CNCF App Delivery SIG (special interest group) that aims to define a vendor-neutral meaning of GitOps. As a first step, they started the OpenGitOps Project.

In a bid to bring consistency to GitOps, the GitOps Working Group has laid out the fundamental principles that govern GitOps. This helps establish a foundation for collaboration and interoperability between the open-source community, vendors, and end-user organizations. Let’s look at what the 4 core principles of GitOps are as defined by the OpenGitOps project.

Links:

• GitOps Working Group
• The OpenGitOps Project

What are the core principles of GitOps?

After much debate and discussion, the OpenGitOps project was able to distill GitOps down to these 4 principles that are succinct, and yet accurately capture the key ideas of GitOps.

#1. Declarative: The entire system has to be described declaratively

With GitOps, Kubernetes is just one example of many modern cloud-native tools that are “declarative,” and that can be defined as code. When we say “declarative,” we mean that the configuration is guaranteed by a set of facts and not by a set of instructions. With your application’s configurations defined declaratively, you can rely on Git as a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be reproduced quickly.

#2. Versioned and immutable: The canonical desired system state is versioned in Git

With the declaration of your system stored in a version-controlled system, and serving as your canonical source of truth, you have an ultimate repository from which everything is derived and driven. This trivializes rollbacks — you can use a ‘Git revert’ to go back to your previous application state anytime you want. With Git’s excellent security assurances, you can also use your SSH key to sign all commits that enforce strong guarantees about the authorship and provenance of your code. The important thing about this version store is that it is immutable — in other words, it doesn’t change over time. This is great for establishing an audit trail.

#3. Pulled automatically: Approved changes are automatically applied to the system

Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make this happen. With GitOps, you have access to a segregated environment outside the defined state. This allows you to separate what you do and how. Of course, there are times when you may want to include a manual review step before certain deployment types, and GitOps allows for this. However, the goal is to have changes go from Git to Kubernetes clusters without being touched or reviewed by humans if it passes all automated tests and checks. Policies are essential to enable such an automated deployment process.

#4. Continuously reconciled: Software agents to ensure correctness and alert on divergence

Once the state of your system is declared and version-controlled, software agents can inform you whenever reality diverges from your expectations. The use of agents also ensures that your entire system is capable of self-healing. And that goes beyond just nodes or pods failure (which can be directly handled by Kubernetes) to fixing human errors. In such situations, software agents act as feedback and control loops for your operations.

Links:

• GitOps: Operations by Pull Request
• The GitOps Pipeline - Part 2

The platform model enables a self-service developer experience

According to the 2021 State of DevOps report, DevOps has matured to a model of stream-aligned teams (Dev), and platform teams (Ops). GitOps encourages this model by enabling platform teams to build and manage an internal platform that is used by application development teams. The platform team creates templates that define resources such as cloud compute, storage, and more. These templates are accessed by developers when they are ready to deploy code to production. Developers select a pre-approved template and deploy their new code according to the configuration in that template. This way, developers can focus on writing code, and not on deploying it, while Ops (or the platform team) can rest assured that all deployed code runs on infrastructure that is pre-approved by them.

GitOps is Continuous Delivery meets Cloud Native

GitOps builds and iterates on ideas drawn from DevOps and Site Reliability Engineering, that started with Martin Fowler’s comprehensive Continuous Integration overview in 2006.

Freedom to choose the tools you need

As a workflow for CI/CD pipelines GitOps has been described as the holy grail of development processes.  Because there is no single tool that can do everything required in your CICD pipeline, GitOps gives you the freedom to choose the best tools for the different parts. You can select a set of tools from the open source ecosystem or from closed source or depending on your use case, you may even combine them. The most difficult part of creating a CICD pipeline is gluing all of the pieces together.

Whatever you choose for your delivery pipeline, applying GitOps best practises with Git (or any version control) should be as an integral component of your process. Doing so will make the transition to continuous delivery easier. This is true not only from a technical point of view but also from a cultural perspective.

“GitOps is game-changing for the industry. It is a replicable, automated, immutable construct where your change management, everything happens in Git.” — Nicolas Chaillan, Chief Software Officer of the U.S. Air Force

Git enables infrastructure as code (IAC) tools

Kubernetes is just one example of many modern cloud-native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions, for example, “there are ten Redis servers”, rather than “start ten Redis servers, and tell me if it worked or not”.

With declarative tools, your entire set of configuration files can be version controlled in Git. By using Git as the source of truth, your apps are more easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can be dependably and quickly reproduced all from Git.

IAC tools vs. GitOps

Infrastructure as Code tools that can provision servers on demand have existed for quite some time. These tools originated the concept of keeping infrastructure configuration versioned, backed up and reproducible from source control.

But now with Kubernetes being almost completely declarative, combined with the immutable container, it is possible to extend some of these concepts to managing applications and their operating system as well.

The ability to manage and compare the current state of both your infrastructure and your applications is what encompasses the GitOps philosophy and its best practices. Adopting GitOps allows you to test, deploy, roll back and roll forward code with a complete audit trail—all from Git. This is possible because Kubernetes is managed almost entirely through declarative configuration and because containers are immutable.

Learn more about how infrastructure as code tools differ from GitOps in the GitOps FAQ

What if my system diverges from the source of truth?

Declarative provisioning tools let you describe your desired true state in Git.  But they suffer from the problem that what is “really true right now” is in the live system, and that may differ from what is described in source control.  

• How do you know if the live system has converged to the desired state?  
• Can you get notified when this differs?  
• What is the “canary in the coal mine” that informs you when you’re in trouble?  
• How do you trigger convergence between the cluster and source control?

IaC tools like Chef, Puppet and Ansible support features like “diff alerts”. These help operators understand when action may need to be taken to “converge” the live system to the intended state (as defined by the configuration scripts). And more recently, the best practice is to deploy immutable images (eg. containers) so that divergence is less likely.

In the “GitOps” model, we use Git to solve for divergence and convergence, aided by the Flux agent that compares the intended state with the actual state.

Works with IAC tools

GitOps takes full advantage of the move towards immutable infrastructure and declarative container orchestration. In order to minimize the risk of change after a deployment, whether intended or by accident via “configuration drift” it is essential to maintain a reproducible and reliable deployment process.

When you apply GitOps principles to “everything”, including machine configuration, applications, and services in addition to alerting rules and dashboards, all are kept under source control. GitOps allows you to define policies that govern each of these components. These policies can be reviewed by a software tool, and Access to the running system should not be required except via Git. Any group of changes may be applied atomically and diffed accordingly. The Git record is then not just an audit log but also a transaction log that you can use to roll back and forth to any snapshot.

Links: 

• Weaveworks & AWS; How we manage Kubernetes clusters
• GitOps for Kubernetes: A DevOps Iteration Focused on Declarative Infrastructure
• What is GitOps Really?

A DIY GitOps Pipeline

A GitOps pipeline is made up of a number of components. Here are the tools most commonly used to build a GitOps pipeline:

• Git hosting service: GitHub, GitLab, BitBucket
• Container registry: AWS ECR
• Build server: Jenkins
• GitOps agent: Weave GitOps, Flux 
• Kubernetes configuration automation: Helm, Kustomize
• Policy as code: Weave GitOps, OPA
• Service mesh: Istio, Linkerd
• Progressive delivery: Flagger
• Container orchestration: Kubernetes
• Monitoring: Prometheus, Grafana

Figure: A GitOps Pipeline

This is just a basic list. In reality, your pipeline could look longer than this depending on your objectives. While this list may look daunting at first, there are readily available solutions that bring these tools together in a convenient way so you can get started quickly. Let's look at these options

Pull vs. Push Pipeline

Most CI/CD tools available today use a push-based model. A push-based pipeline means that code starts with the CI system and may continue its path through a series of encoded scripts or uses ‘kubectl’ by hand to push any changes to the Kubernetes cluster.

The reason you don’t want to use your CI system as the deployment impetus or do it manually on the command line is because of the potential to expose credentials outside of your cluster. While it is possible to secure both your CI/CD scripts and the command line, you are working outside the trust domain of your cluster. This is generally not good practice and is why CI systems can be known as attack vectors for production.

Typical push pipeline with read/write permission outside of the cluster:

Figure: Typical Push Pipeline

In Weave GitOps, images are pulled and credentials are kept inside the cluster:

Figure: Pull-based pipeline

The majority of the CI/CD tools accessible to us are based on a push-based model where the code begins with the CI system and goes through a series of encoded scripts or by manually using ‘kubectl’ to push changes to the Kubernetes cluster. But using one’s CI system as the deployment impetus isn’t always considered a good practice because you are working outside your trust domain.

On the other hand, Weave GitOps’ pull pipeline uses two strategic components: a Deployment Automator responsible for watching the image registry and a Deployment Synchronizer residing in the cluster to maintain its state. In the middle of it all, we have a single source of truth for manifests.

Figure: Weave GitOps pull pipeline

Links:

• How secure is your pipeline?

The pull pipeline

Weave GitOps is a suite of open-source continuous delivery tools designed to run apps in any Kubernetes cluster. In the backend, it is powered by Flux, which is the leading GitOps agent that Weaveworks developed and donated to the CNCF. Apart from Flux, it also has easy integrations to all the most commonly used GitOps tools. Weave GitOps is quite easy to use and efficient when it comes to setting up a GitOps pipeline.

Weave GitOps’ Pull Pipeline

Weave GitOps uses a pull strategy that consists of two key components: a “Deployment Automator” that watches the image registry and a “Deployment Synchronizer” that sits in the cluster to maintain its state.

At the center of our pull pipeline pattern is a single source of truth for manifests (or a config repo). Developers push their updated code to the code base repository; where the change is picked up by the CI tool and ultimately builds a Docker image. The Weave GitOps ‘Deployment Automator’ notices the image, pulls the new image from the repository, and then updates its YAML in the config repo. The Deployment Synchronizer then detects that the cluster is out of date,  pulls the changed manifests from the config repo, and deploys the new image to the cluster.

Weave GitOps Enterprise

Weave GitOps Enterprise takes Weave GitOps to the next level. It has all the built-in benefits of GitOps from the open-source version and adds many capabilities that are meant to scale GitOps at the enterprise level. It makes Kubernetes easier to operate and empowers your team to deliver fast and secure cloud-native applications. With it, you can automate platform configuration using policies, and even roll out similar cluster stacks to different environments.

Weave GitOps Enterprise addresses day 2 operation problems. Its sophisticated cluster lifecycle management handles every phase from bootstrapping to maintenance, upgrades, and patches, all the way up to large-scale cluster fleet management spread across multiple cloud platforms and on-premise establishments.

Weave GitOps Enterprise provides a fully-featured GitOps system along with integrated observability and logging functionalities, GitOps runtime, and a policy engine.

Key benefits of GitOps

Automated delivery pipelines roll out changes to your infrastructure when changes are made to Git. But the idea of GitOps goes further than that – it uses tools to compare the actual production state of your whole application with what’s under source control and then it tells you when your cluster doesn’t match the declared state in Git.

Figure: GitOps – A Git-centric framework for continuous deployment

By applying GitOps best practices, there is a pre-defined, ultimate source of truth for both your infrastructure and application code that lets your development teams increase velocity and improve system reliability.

To that end, the benefits of applying GitOps best practices are far-reaching as it provides:

1. Increased productivity

Continuous deployment automation with an integrated feedback control loop speeds up ‘Mean Time to Deployment’ (MTTD). Teams can ship 30-100 times more changes per day, increasing overall development output by 2-3 times.

2. Enhanced developer experience

With GitOps, you get to push code and not containers. Developers can use familiar tools like Git to manage updates and features of Kubernetes more rapidly without having to know its internal workings. Even the newly onboarded developers can get quickly up to speed and be productive within days instead of months. All this is possible because of the platform model, and the self-service developer experience that GitOps enables.

3. Improved audibility

When you use Git workflows to manage your cluster, you automatically gain a convenient audit log of all cluster changes outside of Kubernetes. An audit trail of who did what, and when to your cluster can be used to meet SOC 2 compliance and help you ensure stability.  

4. Higher reliability

With Git’s capability to revert and fork, you gain stable and reproducible rollbacks. Moreover, since your entire system is described in Git, you also have a single source of truth from which you can recover after a meltdown, reducing your mean time to recovery (MTTR) from hours to minutes.

5. Consistency and standardization

Since GitOps gives you one model for making infrastructure, apps, and Kubernetes add-on changes, you have consistent end-to-end workflows available across your entire organization if required. Not only are your continuous integration and continuous deployment (CI/CD) pipelines all driven by pull requests, but your operations tasks are also fully reproducible through Git.

6. Stronger security guardrails

Git offers strong correctness and security guardrails backed by strong cryptography used to track and manage changes. That, along with the ability to sign the changes to prove authorship and origin, is key to a secure definition of the desired state of the cluster.

7. Faster development

By adopting GitOps best practices developers use familiar tools like Git to manage updates and features to Kubernetes more rapidly. By continuously pushing feature updates, businesses are more agile, can respond more quickly to customer demands, and are more competitive in the marketplace.

8. Better ops

With GitOps, you have a complete end-to-end pipeline. Not only are your continuous integration and continuous deployment pipelines all driven by pull requests, but your operations tasks are also fully reproducible through Git. Policies allow for endless customization of your processes while not compromising on security and pace of operations.

9. Stronger security guardrails

Git’s strong correctness and security guarantees, backed by the strong cryptography used to track and manage changes, as well as the ability to sign changes to prove authorship and origin is key to a correct and secure definition of the desired state of the cluster. This along with the automated policy-based reviews of each step make GitOps a great way to secure software delivery pipelines.

If a security breach does occur, the immutable and auditable source of truth can be used to recreate a new system independently of the compromised one, reducing downtime and allowing a much better incident response.

Separation of responsibility between packaging software and releasing it to a production environment also embodies the security principle of least privilege, reducing the impact of compromise and providing a smaller attack surface.

10. Easier compliance and auditing

Since changes are tracked and logged in a secure manner, compliance, and auditing are made trivial. The use of GitOps agent Flux allows you to compare a trusted definition of the state of the cluster with the actual running cluster, ensuring that the tracked and auditable changes match reality.

The State of DevOps report released by Puppet every year gives us a near-accurate pulse of the DevOps movement at that time. The report from 2021 highlights the importance of culture and the need to clearly define roles and responsibilities by using the platform approach. GitOps is the best way to meet both of these objectives.

Ideas like self-service developer platforms, policy-as-code, progressive delivery, and on-premise GitOps are set to bring revolutionary developments in the deployment and management of applications and infrastructure. Whichever direction GitOps progresses in, Weave GitOps will embody the latest and most effective GitOps ideas into an easy-to-use solution. Try it out for yourself with Weave GitOps.

Links:

• Weave GitOps
• Weave GitOps Enterprise
• GitOps: High-velocity CICD for Kubernetes
• How to Boost Business Performance with GitOps: The Facts
• Multi-cloud Strategies with Kubernetes and GitOps

Why Weaveworks?

Weaveworks is the creator of Weave GitOps, a continuous delivery product for running and managing apps running in Kubernetes. Among its numerous attributes, it has simplified deployment monitoring and management for containers and microservices, and it extends and complements popular orchestrators, allowing developers and DevOps teams to make faster deployments, insightful monitoring, visualization, and networking.

We are also the creators and maintainers of CNCF OSS GitOps projects like Flux and Flagger which are trusted by thousands of customers to help with their unique use cases. 

But more than that, Weave GitOps has its own diverse set of use cases that are sure to solve fundamental challenges you face at your organization:

• Continuous application delivery at cloud-native speed
• Accelerating software lifecycles through DevOps automation
• Trusted application delivery
• Managing hybrid and multi-cloud complexities
• Progressive delivery
• Boosting collaboration, reliability, and autonomy through self-service

Additionally, we are an AWS Strategic Partner, helping them accelerate EKS adoption. Weaveworks also created Eksctl which today is an indispensable CLI tool for providing managed Kubernetes services for EC2. Furthermore, Weaveworks is a key partner with Microsoft Azure where it provides support for Flux in Microsoft GitOps deployments.

Need help?

With Weave GitOps, you can accelerate your path to production-ready Kubernetes, and get a team of cloud-native experts on your side to train your team to adopt GitOps.

Get started with Weave GitOps

Request a Demo

---

With Weave GitOps, you can accelerate your path to production-ready Kubernetes, and get a team of cloud-native experts on your side to train your team to adopt GitOps.

Get started with Weave GitOps