Consistent Observability Across Clusters

By Steve Waterworth

June 16, 2022

Self-Service

Hybrid & Multi Cloud

This blog demonstrates how to build an observability stack with Weave GitOps. We will be using Prometheus, Loki, Promtail, Grafana and Helm and of course GitOps to simplify the manual and labour intensive set up. Break free from the drudgery with GitOps Automation.

Observability for GitOps pipelines

Improve Kubernetes operational intelligence with multi cluster observability

Beginner’s Guide to Platform Engineering: Simplifying Software Delivery

The CNCF 2021 survey has some interesting metrics.

85% have 5+ clusters.
67% have 20+ clusters.
93% have 1+ production clusters.
64% have 5+ production clusters.

The CNCF concludes that Kubernetes has crossed the chasm and is now mainstream.

With the majority of organizations having multiple Kubernetes clusters, maintaining consistency across them all is a heavy burden on the platform team; if it’s done manually. Using Weave GitOps will significantly reduce the effort required. It can be used for any application stack deployed to Kubernetes but this article will show an approach for an Observability stack.

The Stack

Keeping with the CNCF theme, the stack that will be deployed consists of:

Prometheus - time series metrics
Loki - log indexing
Promtail - log shipper
Grafana - dashboard manager
Dashboards - custom dashboards for Grafana

That’s quite a lot of moving parts, imaging installing and configuring that lot manually, multiple times across 20+ clusters. The definition of drudgery?

Fortunately, some of the work has already been done for you because most of these are available as Helm charts. However, there’s still plenty to be done to keep everything up to date and consistent across multiple clusters.

Weave GitOps to the Rescue

Flux has some Custom Resource Definitions (CRD) to handle Helm:

HelmRepository - create these first
HelmRelease - references HelmRepository

Here’s an example for the Kubernetes Prometheus stack.

$ flux create source helm prometheus-community \
    –url=https://prometheus-community.github.io/helm-charts \
    –interval=1h

This creates a HelmRepository which will be polled hourly, the shell equivalent is:

$ helm repo add prometheus-community \
     https://prometheus-community.github.io/helm-charts
$ while true; do; helm repo update; sleep 3600; done

Next the chart is added from the repository.

$ flux create hr prometheus-stack \
    –interval=1h \
    –source=HelmRepository/prometheus-community \
    –chart=kube-prometheus-stack \
    –chart-version=”34.9.1” \
    –target-namespace=prometheus \
    –values=values.yaml

This creates the installation of the chart. The shell equivalent is:

$ helm install \
     prometheus-stack \
     prometheus-community/kube-prometheus-stack \
     –version=”34.9.1” \
     –namespace=prometheus \
     –values=values.yaml \

The difference between the two commands is the interval flag. With the chart version pinned, this has no effect. However, semver syntax is supported for the version flag. Changing the version, for example, to “34.9.x”, Weave GitOps will poll the chart every hour and automatically apply the latest patch. That’s one task removed from the burden of manual patching.

The Prometheus Community Helm chart includes Grafana, Kube State Metrics and Node Exporter. Use the same technique to add the Grafana Helm Repository then the Loki and Promtail charts. However, before you start typing read on, there’s more to come.

Dashboards

The Prometheus Community Helm chart includes some standard dashboards for Kubernetes resources. You will want to add your own dashboards to keep an eye on your applications. The Grafana configuration included in the Helm chart will automatically pick up dashboard definitions from ConfigMaps with an appropriate label.

One option is to manage your custom dashboard definitions inside a Git repository and have Weave GitOps apply them to the clusters.

Create the Git repository first.

$ flux create source git acme-dashboards \
    –url=ssh://git@github.com/acme/dashboards \
    –branch=main \
    –interval=1h \
    –secret-ref=ssh-credentails

Next create the application that contains the dashboard ConfigMaps

$ flux create kustomization app-dashboards \
    –target-namespace=monitoring \
    –source=acme-dashboards \
    –path=kustomize/app-dashboards \
    –prune=true \
    –interval=1h

Using the Kustomize ConfigMap generator for the dashboards saves a lot of work.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: kustomization
configMapGenerator:
   - name: app-dashboards
     files:
          - dashboards/app-overview.json
          - dashboards/cart-service.json
          - dashboards/search-service.json
      options:
        labels:
          grafana_dashboard: “1”

The ConfigMap generator automatically builds a ConfigMap with the specified files embedded. The label ensures that Grafana picks them up. To add a new dashboard, design it in Grafana, export it to a JSON file and add it to the Kustomization. In Git merge the change into main and the dashboards are automatically deployed across your clusters by Weave GitOps.

Helm Wrapper

That’s still a lot of individual steps and quite a few GitOps resources created. Helm charts can be recursive, a chart can include other charts, which can include charts…

Rather than managing all the components separately, a better approach is to build your own Helm chart that includes all the components you need and manage it in Git, of course. Unfortunately Helm does not have the equivalent of Kustomize ConfigMap generator therefore we’ll need a two step build process: generate the ConfigMap with Kustomize then package it all up with Helm.

The main components of a Helm chart are a Chart.yaml file, templates directory and a values.yaml file. For this example the Chart.yaml would be:

apiVersion: v2
name: observability
version: 1.1.0
description: Observability stack
dependencies:
  - name: kube-prometheus-stack
    version: 34.10.0
    repository: https://prometheus-community.github.io/helm-charts
  - name: loki
    version: 2.11.0
    repository: https://grafana.github.io/helm-charts
  - name: promtail
    version: 4.2.0
    repository: https://grafana.github.io/helm-charts

This pulls in the required dependencies. Before the chart gets packaged up, Kustomize is used to build the ConfigMap of all our custom dashboards.

$ kustomize build ./kustomize > templates/dashboards.yaml

Now when the chart is packaged up it will include all the dependencies and all our custom dashboards. The packaged chart can be hosted on any HTTP server that’s reachable by your clusters. Weave GitOps supports using various types of authentication when accessing Helm repositories.

Quite Easily Done

Now we have a central place, fully version controlled and audited by Git where our Observability platform can be managed from. At any point we can make changes to our Helm chart to add or modify dashboards or patch any of the dependencies. The version of our chart is incremented then it’s packaged and published. Weave GitOps will automatically pick up the new version and deploy it across all your clusters. Weave GitOps makes it easy for DevOps engineers to ensure that applications are consistently deployed across multiple Kubernetes clusters. Break free from drudgery with Weave GitOps.