Amazon EKS Anywhere Multi-Cluster Management with Weave GitOps Enterprise

By Weaveworks

August 21, 2023

Weave GitOps Enterprise

AWS

Scale your Kubernetes clusters effortlessly with GitOps on Amazon EKS Anywhere. Learn how to maintain consistency, enforce policies, and manage multi-cluster deployments in this comprehensive blog by AWS and Weaveworks.

Challenges with Multi-Cluster Management

Managing a single production Kubernetes cluster can be a complex undertaking. In reality, more often than not you will need to run multiple Kubernetes clusters because of application security, software licensing requirements, high availability, and disaster recovery purposes.

The management of multiple Kubernetes clusters can be challenging, especially for platform teams. Any deployment at scale will need automation in order to achieve consistency across clusters and clouds.

For successful multi-cluster operation, teams need to establish proficiency in these 4 key areas:

Automated, consistent way to create and upgrade clusters.
Recorded history of configuration changes to cluster.
Governance controls that provide a reliable way to ensure only approved changes are being rolled out.
Ability to deploy changes to multiple clusters at once without needing to run commands repeatedly.
GitOps for Cluster Management

Let’s start with defining GitOps, an increasingly popular model for infrastructure management.

GitOps can be summarized as two things:

It is an operating model for Kubernetes and other cloud-native technologies, a set of best practices bringing together Git deployment, management, and monitoring for containerized clusters and applications.
It offers a path towards a self-service developer experience for managing applications and automating CI/CD pipelines, unifying development and operations teams.

In other words, GitOps is an operational model offering customers the ability to manage the state of multiple Kubernetes clusters by extending the best practices of version control, immutable artifacts, and automation. Most importantly, defining and managing your clusters and platform in GitOps enables you to scale your platform without increasing your engineering team.

GitOps utilizes version control, so you can reap the same benefits that you love for your application code lifecycle for your Kubernetes platform. With that in mind, GitOps has become the most effective and risk-free way to manage Kubernetes at scale and that of course also applies to Kubernetes clusters running on EKS.

The leading GitOps software is Weave GitOps Enterprise as it provides teams with a fully integrated suite of GitOps tools that enhance management and control of cluster lifecycle and management.

Cluster Lifecycle Management

After the mantra “cattle, not pets,” GitOps simplifies managing cluster lifecycle at scale - even massive scale. Through Git pull requests, every action is recorded and auditable, which allows small teams to create, update and delete clusters across whole fleets. In short, it makes things harder to break and easier to recover.

Built for a multi-cluster world

An increasing number of enterprises today are seeking out the feature and cost advantages of multi-cloud - meaning they maintain infrastructures not only on-premise with EKS-A , but also Amazon EKS in the cloud. Weave GitOps Enterprise is designed to support platform and fleet management across on-prem, hybrid and multi-cloud infrastructures.

Accelerate your EKS Adoption with Weave GitOps

AWS and Weaveworks partner on technical advancements, especially for the EKS product suite. If you want to accelerate your EKS adoption and automation with GitOps - we can get you to a well-architected platform in just a few weeks.

LEARN MORE

EKS Anywhere Multi-cluster Management

EKS Anywhere provides the ability to define your clusters as either a management cluster or a workload cluster. This specifically makes it easier for you to manage multiple clusters.

The management cluster is essentially a long-standing cluster that can be used to modify, upgrade, and delete workload clusters. Separating management features into a separate, persistent management cluster provides a cleaner model for managing the lifecycle of workload clusters (to create, upgrade, and delete clusters), while workload clusters run user applications.

This approach also reduces provider permissions for workload clusters. While it is certainly possible to run standalone workload clusters, creating an EKS Anywhere management cluster is the recommended model when the need is to run and manage more than a few workload clusters.

EKS Anywhere and GitOps Automation

EKS Anywhere also supports GitOps workflows for managing clusters. While Gitops is widely used for application deployments today, EKS Anywhere provides the option to apply the same principles for effectively managing the lifecycle of your Kubernetes clusters. When you create a cluster with GitOps enabled, EKS Anywhere will automatically commit your cluster configuration to the provided GitHub repository and install a GitOps toolkit on your cluster which watches that committed configuration file. You can then manage the scale of the cluster by making changes to the version controlled cluster configuration file and committing the changes. Once a change has been detected by the GitOps controller running in your cluster, the scale of the cluster will be adjusted to match the committed configuration file.

Apart from providing a version controlled mechanism for change management across your multiple clusters, this also provides you a way to create and enforce approval and governance controls for any changes being made to your cluster via controlling the commits to your Git repository. The Weave GitOps Flux CD Controller integrated with EKS Anywhere currently provides the ability to manage a subset of cluster properties with GitOps - here is a list of supported properties.

Recommended Repo structure for Multi-Cluster Management

When deploying multiple clusters it is important to have a definition for each cluster separated into its own directory structure so that individual clusters can be managed separately. As the number of clusters increases, it becomes increasingly difficult to manage and keep all the clusters up to date. Hence we recommend grouping clusters so that changes can be applied to groups of clusters at once.

The minimum grouping of clusters would be by environment, such as development, staging, and production. This allows changes to be rolled out to each environment in turn to check the result of the upgrade prior to applying the change to the next environment.

For cluster management of a large number of clusters, this may not be sufficient, and additional groups may be required to subdivide the environments further or, in fact, provide additional groupings based on tenants, locations, customers or any other grouping which makes sense to the organizational requirements.

An example of a repository structure is below:

clusters/ 
 ├─ bases/
│    ├─ rbac.yaml
├─ dev-bases/
│    ├─ sources.yaml 
│    ├─ platform.yaml
├─ stg-bases/
│    ├─ sources.yaml
│    ├─ platform.yaml
├─ prd-bases/
│    ├─ sources.yaml
│    ├─ platform.yaml
├─ devteam1/
│  ├─ bases/
│    ├─ config.yaml
│ ├─ dev/
│     ├─ clusters-bases-kustomization.yaml
│     ├─ clusters-dev-bases-kustomization.yaml
│     ├─ clusters-devteam1-bases-kustomization.yaml
│     ├─ profiles.yaml
│     ├─ flux-system.yaml
│     ├─ apps.yaml
│ ├─ staging/
│     ├─ clusters-bases-kustomization.yaml
│     ├─ clusters-stg-bases-kustomization.yaml
│     ├─ clusters-devteam1-bases-kustomization.yaml

 ├─ profiles.yaml
│     ├─ flux-system.yaml
│     ├─ apps.yaml
│ ├─ prod/
│     ├─ clusters-bases-kustomization.yaml
│     ├─ clusters-prd-bases-kustomization.yaml
│     ├─ clusters-devteam1-bases-kustomization.yaml
│     ├─ profiles.yaml
│     ├─ flux-system.yaml
│     ├─ apps.yaml

In the above example, if you need to roll out a new application to the devteam1 collection of clusters then you can either install the new files into the individual cluster directories, or use the bases directory for one of the groupings, such as clusters/dev-bases to roll out to all the dev clusters or the clusters/devteam1/bases directory to roll-out to all devteam1 clusters at once.

Recommended Repo Structure for App Deployment Across Multi-Cluster

One option we recommend is to use directories for the different environments/clusters. Below is an example of how to structure your repository and the directories for an application across three different environments (dev, staging, prod).

environments/
├─ dev/
│ ├─ app1-helm-repo.yaml
│ ├─ app1-helm-release.yaml
│ ├─ app1-ingress.yaml
├─ staging/
│ ├─ app1-helm-repo.yaml
│ ├─ app1-helm-release.yaml
│ ├─ app1-ingress.yaml
├─ prod/
│ ├─ app1-helm-repo.yaml
│ ├─ app1-helm-release.yaml
│ ├─ app1-ingress.yaml 
... (application code)

Risks and Pitfalls of multi-cluster management and apps deployment with GitOps

Managing a strong release process is the most important consideration when dealing with scaled GitOps deployments. Rolling-out changes across many clusters at once can lead to unpredictable consequences due to lack of testing. It is incredibly important to make sure that changes are versioned and rolled out to groups of clusters in a controlled manner. It allows the changes to be fully tested in each environment or cluster group before moving on to the next environment or group. You can also catch errors and configuration issues in the environments prior to production to prevent issues reaching the production environment that then impacts customers and the business.

Many naive implementations of GitOps do not address this point and result in cluster and platform apps stability issues for app developers who use the cluster and platform apps as a base for their code. A stable cluster platform is absolutely essential for application developers to trust the platform and use it efficiently and effectively.

Security Best Practices

Let’s also have a look at governance best practices to avoid the dreaded 2 am wake-up call. Here are 3 best practices that can be implemented immediately and keep your clusters more secure.

1.Use Policies to Enforce Governance

One crucial step to remove DevOps friction is efficient governance through embedded policies. To prevent potential misconfigurations or errors in production, we recommend setting up policy admission controllers with Weave Policy Engine and Weave GitOps Enterprise right at the root of trust. Developer autonomy and self-service workflows are preserved while operations teams can rest assured that security guardrails are in place.

2. Apply Least Privilege Practices in your Kubernetes Clusters

Building on the previous idea of policies, we strongly recommend never granting more access to users than absolutely required. Making errors is only human, but having those slip into production might not only cause a big headache but can be costly. In practice limiting the blast radius and quiet noisy neighbors serves organizations and teams well. For example, avoid ClusterRoleBindings in Kubernetes, don’t use wildcards but do use tainting, affinity, and anti-affinity for core and key services.

3. Reduce the Blast Radius with effective Secret Management

Secrets are helping teams to protect sensitive credentials. For example, authentication details to get images from a registry. Secrets management is an extremely effective and easy to accomplish task when reducing the blast radius. We advise storing secrets in AWS Secret Manager or another secret storage solution and using External Secrets Operator to rotate secrets successfully.

Weave GitOps Enterprise and EKS Anywhere

Weave Gitops Enterprise runs as a management cluster and is able to deploy many EKS-Anywhere clusters to multiple providers in on-premise environments, such as VMWare, Openstack Bare Metal, or Tinkerbell for Bare Metal.

Weave GitOps Enterprise acts as the management cluster for any leaf clusters that are deployed on-premise or just EKS clusters in the cloud using the same tools. You can deploy fleets of clusters with platform applications and host many different tenant applications on top.

The user interface allows teams to view clusters and Gitops objects, their sync status and health in a single pane of glass view. Platform teams can view their fleet of clusters, quickly uncover errors and effectively troubleshoot across the whole fleet. Application teams can view clusters and applications in the namespaces they have access to and observe deployments of these applications across multiple environments.

Weave GitOps Enterprise provides templates for many Kubernetes objects. Platform teams have access to a full catalog of templates that can be used to build a self-service interface for application developers. Using templates in the UI or the CLI or even in an IDE such as VSCode, allows application developers to set up and deploy applications across fleets of clusters and in combination with EKS-Anywhere, even deploy to on-premise environments.

Best Practices for Hybrid Cloud Kubernetes with EKS and Weave GitOps

Download our whitepaper to discover how Amazon EKS and Weave GitOps streamline and scale application deployment and management in a hybrid cloud environment.

DOWNLOAD NOW