Safety Fast with Weave GitOps Trusted & Progressive Delivery

By Steve Waterworth
September 15, 2022

Ship new features and fixes faster with Weave GitOps Trusted Delivery as the safety net. Trusted Delivery uses Policy as Code to check manifests throughout the SDLC combined with progressive delivery.

Related posts

September Release - Weave GitOps 2022.09

Shifting Security Left with GitOps and Trusted Delivery

Secure your CI/CD pipeline with Trusted Delivery

Building better code faster is one of the common goals of DevOps teams. The DORA “State of DevOps Report” states “excellence in software delivery and operational performance drives organisational performance” or in other words the speed at which an organisation can ship new features and fixes has a direct impact on business performance.

With increased speed of software delivery comes increased risk of something going wrong; as the saying goes “More haste less speed”. Humans are not very good at repetitive tasks at the best of times and the added pressure of increased speed only compounds the situation. Increased automation mitigates the risk, computers excel in performing repetitive tasks with 100% accuracy.

GitOps is the foundation layer of automation for rapid software delivery. It provides control via the familiar Git workflow of pull request, review, approve and merge. It provides a complete audit trail of every change enabling swift rollback to a previous known good state in the event of a suboptimal deployment. It eliminates configuration drift and performs consistent updates of application environments.

Safety Net with Trusted & Progressive Delivery

An application microservice consists of two main components, the code and the deployment manifest. Both of these components need to be robust and error free. The challenge is in ensuring that this is true without slowing down the software delivery process. Unit tests and integration testing during promotion through test and staging environments catch most coding problems but what about ensuring the deployment manifests are secure, resilient and follow coding standards?

Securing Deployment Manifests with Policy as Code

A set of deployment manifest files can easily be a few hundred lines of YAML. There are numerous ways that this configuration can reduce the security, resilience, and maintainability of the microservice. Weave GitOps provides a curated library of over one hundred policies that check the configuration against well known best practices and industry standards such as: HIPAA, PCI-DSS, GDPR, SOC II, MITRE ATTACK, etc. The policies are built on Open Policy Agent (OPA) making it easy to add your own corporate policies. Policy violations can prevent the merging of a pull request, building of a container image. Ultimately the Policy Engine admission controller prevents out of policy manifests making it into the cluster.

Weave GitOps can check for policy compliance at multiple points during the software lifecycle: commit, build, deploy and runtime. Each policy is an instance of a Custom Resource Definition (CRD), just like all Kubernetes entities it’s a YAML file. The active policies are managed via Git and policies can be selected by branch and/or by directory, with or without Kustomize. Alternatively your active policy libraries could be maintained as Helm Charts. This enables central management of your policy libraries and the ability to apply different sets of policies to different environments.

With Weave GitOps Policy Engine continuously checking for policy compliance, it provides the confidence to deliver new features and fixes at speed.

De-Risking Deployments with Progressive Delivery

Safety_Fast_with_Weave_GitOps_Trusted_&_Progressive_Delivery1.png

New features and fixes will be promoted through different environments getting tested at each step. No matter how rigorous the testing performed in Dev, Test and Staging environments, Murphy’s Law clearly states that the best way to break software is to deploy it into Production. By default Kubernetes provides a rolling deployment strategy where one by one, the Pods for the target microservice are terminated and replaced with the new version. This typically only takes a few minutes after which the new version is receiving 100% of the traffic. In the event of a failure 100% of your customers will be affected.

Safety_Fast_with_Weave_GitOps_Trusted_&_Progressive_Delivery2.png

With a Canary Release, the new version initially only receives a small percentage of the load. This percentage is gradually increased as long as its KPIs remain healthy, until it ultimately receives 100% of the load. In the event of the KPI health check failing at any point during the Canary Release, it is rolled back to the previous good version.

Using a Canary Release strategy mitigates the risk of a new release. Even if it’s a total disaster, only a small number of users will be affected for a short period as the rollback is executed quickly. However, there is an extra burden in configuring Canary Releases and the enabling service mesh that prevents many organisations from adopting this technique.

Weave GitOps automates managing Canary Releases and provides a simple single file configuration, making adoption much easier. A service mesh is optional too, Canary Releases can be facilitated with an ingress controller such as: NGINX, Traefik, Contour, Skipper. Weave GitOps makes implementing progressive delivery easy.

In Action

Policy as Code

This functionality is included along with access to the curated policy library with a Weave GitOps install. You can pick and choose which policies from the library along with any of your own policies to apply to individual clusters. You can use a Git repository directory structure, for example.

policy-library

|- policies

|- clusters

    |- test

    |- staging

    |- production

Each cluster has its own subdirectory containing the policies to be applied. These are applied to the clusters via GitOps reconciliation just like other Kubernetes entities. Alternatively Helm can be used with either one Chart covering all environments using conditional blocks for different environments or separate charts for each environment. Weave GitOps supports OCI registries for Helm Charts allowing you to store the Charts with your container images in the same registry reducing the administrative burden.

Safety_Fast_with_Weave_GitOps_Trusted_&_Progressive_Delivery3.png

That covers the admission controller for deployment time validation and run time reporting. To perform validation at commit and/or build time, configure an action for your platform e.g. GitHub, GitLab, Bitbucket, CircleCI, etc. A Docker image is provided that performs the validation, which can fetch the policy library from Git or Helm.

Under certain circumstances and if enabled it can perform automatic remediation of the violation. As a simple example, a policy may enforce a minimum replica count of two for Deployments. During validation if a breach is detected, a pull request will be created with a suggested fix.

Progressive Delivery

Safety_Fast_with_Weave_GitOps_Trusted_&_Progressive_Delivery4.png

Weave GitOps makes it easy to adopt progressive delivery without requiring to learn the dark arts of service meshes; in fact canary releases can be achieved with just a regular ingress controller such as Nginx. For any service that you require a canary release, just replace the usual Kubernetes Service entity with a Weave GitOps Canary entity.

The Service definition moves inside the Canary definition along with the rollout step strategy and KPI checks. The steps can be linear 5, 10, 15, 20, 25… or nonlinear 5, 10, 20, 50, 80. For KPI checks, Prometheus with default queries are included, alternatively other observability platforms such as: Datadog, New Relic, Cloud Watch, Dynatrace are supported. With Weave GitOps, it’s easy to adopt progressive delivery without the usual steep learning curve.

Safety_Fast_with_Weave_GitOps_Trusted_&_Progressive_Delivery5.png

Safety Fast with Weave GitOps

Manifest validation with Weave GitOps policy as code guarantees that microservice deployments are secure, resilient and conform to your enterprise standards without slowing down the software delivery life cycle. Checks are shifted left eliminating procedural bottlenecks. Easy adoption of progressive delivery with Weave GitOps mitigates the risks involved with rapid production deployments. In the event of a suboptimal release, only a limited number of customers are affected for a minimum amount of time.


To find out how Weave GitOps can accelerate your software delivery life cycle, contact Weaveworks for a demonstration of both Trusted Delivery and Progressive Delivery.

Contact us Today

Related posts

September Release - Weave GitOps 2022.09

Shifting Security Left with GitOps and Trusted Delivery

Secure your CI/CD pipeline with Trusted Delivery

Whitepaper: Progressive Delivery with GitOps

This handy pocket guide goes into details on Progressive Delivery, how it works and how you can get started today.

Download Whitepaper