Multi-cluster Application Deployment Made Easy with GitOpsSets
Weave GitOps Enterprise’s new feature, GitOpsSets, groups application resources into sets, thus simplifying the process of applying configuration updates and tracking changes across environments. Read all about it here.
Further reading
Streamline Kubernetes Workflows with Weave GitOps Assured
Make Your Path to Production Secure & Predictable With Weave GitOps Pipelines
Close the Cloud-Native Skills Gap and Use Automation to Boost Developer Productivity with Weave GitOps Release 2023.04
Create Safe Perimeter Boundaries for Development Teams with Workspaces
GitOpsSets is one of our newest features released in the latest Weave GitOps Enterprise product launch. This feature enables Platform Operators to have a single resource definition for an application for multiple environments and a fleet of clusters. GitOpsSets reduces the complexity of managing multiple applications and resources, streamlines deployment processes, and ensures consistency across your infrastructure.
In this article, we will explore in depth our latest feature, why it’s needed, its benefits, and how you can get started.
The Problem: Cloud-Native Infrastructure ComplexityWith the majority of organizations running numerous clusters hosting multiple applications, managing the configuration of these different environments is a nontrivial challenge faced by most DevOps teams. Let’s just take a moment to consider the complexity involved:
A typical microservices application deployed on a Kubernetes cluster has numerous Pods, and each of those Pods will be at different stages of their delivery pipeline being deployed across development, staging, and production environments. That’s already a lot of moving parts for just one application, and organizations have more than just one application.
The 2021 Datadog Container Use report shows that on average, an organization is running 1000 Pods, and the trend is rapidly rising. Therefore an average Kubernetes estate is running 20 clusters hosting 1000 Pods, with each Pod running at different versions as they are promoted through development, staging, and production. That’s a lot of configuration complexity to keep up with.
Git is the ideal tool to manage the complexity of all these different configurations. A previous blog post provided an overview of the many different ways to structure Git repositories for micro-service deployments across multiple Kubernetes environments. A real-world example of how Zscaler manages its multiple tenant deployments - here is the recorded talk from Zscaler.
Automating Complexity with Weave GitOpsSetsThe pure GitOps approach does make it considerably easier to manage the complexity of deploying multiple applications across multiple clusters, however, there’s still quite a lot of manual work involved. Weave GitOps has automated most of the manual steps with numerous features - such as GitOpsSets - and wrapped it all up with a nice GUI. Let’s take a look.
What are GitOpsSets?
GitOpsSets enable Platform Operators to have a single definition for an application for multiple environments and a fleet of clusters. With a single resource definition, the Ops team can generate environment and cluster-specific configurations. GitOpsSets will create all the objects and Flux primitives that are required to deploy this application successfully. Now Operators no longer need to edit 100’s of files - all of that can now be done with a single command.
GitOpsSets combine Generators and Templates:
Generators: these are dynamic functions that can fulfill a variety of tasks, they generate key/value maps of data that can be injected into templates.
Templates: These are templated YAML files describing Kubernetes resources which are combined with the generated data and applied directly to the cluster.
GitOpsSets supports multiple generators including:
List Generator: The simplest generator which provides a list of Key/Value pairs that you want to feed the template with.
Git Generator: Enable to extract a set of files (environment-specific configurations) from a Flux GitRepository, and make the contents of these available to the templates, this would let you have config in app-dev.json, app-staging.json and app-production.json for example, and the contents of these would be available to the templates.
Matrix Generator: Combine generators into the desired compounded input.
Pull request Generator: Automatically discover open pull requests within a repository to generate a new deployment.
API Client Generator: Poll an HTTP endpoint and parse the result as the generated values.
Cluster Generator: Query for a set of Weave GitOps Clusters matching a set of labels.
Supercharge your Kubernetes adoption with the Assured Accelerator
The Weave GitOps Assured Accelerator program delivers a GitOps enabled Kubernetes platform in 4 weeks on any environment.
Learn moreHow do GitOpsSets work?
Templated resources can be regular Kubernetes entities such as: Deployment, Statefulset, Service, etc. But to really get value with complete automation the Flux entities should be used: HelmRelease, Kustomization, etc.
The example environment consists of a management cluster running Weave GitOps and leaf clusters: Tom, Dick, and Harry. If these clusters were provisioned by Weave GitOps or provisioned manually then joined to the management cluster, the Flux engine will be automatically deployed to them.
Tom stage=staging
Dick stage=production
Harry stage=production
As an example, a HelmRepository and HelmRelease are created and committed to Git, reconciliation ensures they are deployed to the management cluster as templates. Next, using the GitOpsSets the distribution of these resources can be defined. The GitOpsSets includes templates for HelmReleases which can reference the cluster secrets to target remote clusters. Following GitOps principles, the definition for the GitOpsSet is merged into Git and the reconciliation deploys it to the cluster. The scheduler picks up the definition, deploying the resources to the target clusters. The Weave GitOps reconciliation engine on each of the target clusters then picks up those resources, and ultimately the Helm Chart is deployed on the target clusters.
The whole process is driven by GitOps principles. Following on from the example above, if the target cluster selector label is changed via a Git workflow from staging to production, the Helm Chart will be removed from staging and deployed to production; promoting the release. The initial staging deployment would be on the cluster Tom by the label selection. Changing the label selection to production would move the Helm Chart to Dick and Harry.
The target cluster definition supports multiple cluster failover with weighting but that’s beyond the scope of this introductory article.
How can Organizations Benefit from GitOpsSets?
Weaveworks GitOpsSets can significantly benefit organizations focused on managing complex cloud-native infrastructure and applications. By enabling the grouping of resources and applications into sets, it simplifies the process of applying configuration updates and tracking changes across environments.
Consequently, developer teams reap many benefits, including:
Reduces errors and speeds up deployment frequency and lead time by giving developers the ability to quickly review code changes in a short-lived preview environment.
Streamlines and shortens the process of getting changes into environments triggered through a natural pull request-based developer flow.
See the resources created from GitOpsSets and the status for each of those resources centrally in the Weave GitOps GUI.
Simplifies the arduous and complex process of defining repetitive, individual parameters and variables by abstracting them to a single definition, making it fast and easy to configure and deploy multiple environments.
Automates deployments to many sets of targets from a single abstraction, saving teams time and removing errors.
Reduces the number of files in the Git repository, no more wondering “is this file up-to-date with the template that it was rendered from?”
See GitOpsSets in Action
In our latest product launch webinar,Joe Dahlquist, VP of Product Marketing, and James Wilson, Director of Product Management, demonstrated the Pipelines features along with GitOps Sets and GitOps Templates.
Watch the webinar in its entirety or skip to some of the key highlights:
Speaker Introductions [min 2.00]
About Weaveworks [min 2:39]
Weave GitOps [min 3:34]
Product Announcements: [min 5:04]
Weaveworks Solutions [min 5:43]
New in Weave GitOps 2023.04 [min 9:12]
Pipelines [min 10:36]
GitOps Templates [min 11:32]
GitOps Sets [min 12:56]>
Weave GitOps Enterprise Demo [min 14.40 - 31.40]
Q&A [min 34.33]
Streamline and Automate Deployment with GitOpsSets
Brought to you by the GitOps and Flux creators, Weave GitOps Enterprise makes it easy to deploy and manage Kubernetes clusters and applications at scale, in any environment. The single management console automates trusted application delivery and secure infrastructure operations on-premise, in the cloud, and at the edge. In addition to GitOpsSets, Weave GitOps Enterprise boasts a number of other distinct features such as GitOps Run, Pipelines, Multi-tenancy, VS Code Extension, Terraform Controller, with many more to come.
Contact Us for a demo to learn more.
Further reading
Streamline Kubernetes Workflows with Weave GitOps Assured
Make Your Path to Production Secure & Predictable With Weave GitOps Pipelines
Close the Cloud-Native Skills Gap and Use Automation to Boost Developer Productivity with Weave GitOps Release 2023.04
Create Safe Perimeter Boundaries for Development Teams with Workspaces
Whitepaper: Why Self-Service is Key to Developer Productivity
Download this whitepaper to learn how you can enable self-service developer experience with GitOps.
