Continuous Delivery with Weave Flux

By bltc34b82b83b949c23
December 15, 2016

Weave Cloud runs on Kubernetes. Kubernetes give us a suite of tools for managing a running system, but it does have boundaries. In particular, it doesn’t care (and shouldn’t care) where your container images come from. But we do — we have...

Related posts

Empowering Platform & Application Teams: A Closer Look at Weave GitOps Enterprise Features

Kubernetes Security - A Complete Guide to Securing Your Containers

Multi-cluster Application Deployment Made Easy with GitOpsSets

Weave Cloud runs on Kubernetes. Kubernetes give us a suite of tools for managing a running system, but it does have boundaries. In particular, it doesn’t care (and shouldn’t care) where your container images come from. But we do — we have taken great care to build a workflow around code review and continuous integration (CI).

Ultimately we want to go from code, to a tested container image, to a running service. Continuous integration (CircleCI and a bit of scripting, in our case) gets us through those first two steps, to a published container image. How do we make the final step?

We built Flux to link CI with Continuous Deployment (CD)

There are multiple sources of information about what is or should or could be running in your Kubernetes cluster. First, there is Kubernetes itself, which can tell us what is running. Second, there is the configuration, in the form of Kubernetes manifests, which we keep in git. Third, there is the container image registry, e.g., quay.io or Docker Hub, which is usually at the far end of your CI pipeline.

Flux tells you when there’s an upgrade ready

In practice, the problem is in two parts: lack of visibility and error-prone updates.

First of all, how do I tell what is running in my cluster and whether there’s a new image to which to upgrade?

We can see which services are running in Kubernetes, and we can see which images a deployment uses. In Weave Cloud we use the convention that a service has a single deployment associated with it, so combining those, we can see which images are running for a particular service. Using Flux:

<code>
$ fluxctl list-services --namespace=sock-shop
SERVICE                 CONTAINER   IMAGE                           RELEASE  POLICY
sock-shop/cart          cart        weaveworksdemos/cart:0.3.0                   
sock-shop/cart-db       cart-db     mongo                                        
sock-shop/catalogue     catalogue   weaveworksdemos/catalogue:0.2.0              
sock-shop/catalogue-db  catalogue-db  weaveworksdemos/catalogue-db:0.2.0         
sock-shop/front-end     front-end   weaveworksdemos/front-end:0.2.0              
sock-shop/orders        orders      weaveworksdemos/orders:0.3.0                 
sock-shop/orders-db     orders-db   mongo                                        
sock-shop/payment       payment     weaveworksdemos/payment:0.3.0                
sock-shop/queue-master  queue-master  weaveworksdemos/queue-master:0.3.0         
sock-shop/rabbitmq      rabbitmq    rabbitmq:3                                   
sock-shop/shipping      shipping    weaveworksdemos/shipping:0.3.0               
sock-shop/user          user        weaveworksdemos/user:0.3.0                   
sock-shop/user-db       user-db     weaveworksdemos/user-db:0.3.0
</code>

We can also see which images are available for a particular service using Flux.

<code>
$ fluxctl list-images --service=sock-shop/front-end
SERVICE             CONTAINER  IMAGE                                  CREATED
sock-shop/front-end  front-end  weaveworksdemos/front-end                    
                     |   7f511af2d21fd601b86b3bed7baa6adfa9c8c669  18 Nov 16 16:00 UTC
                     |   latest                                    18 Nov 16 16:00 UTC
                     |   snapshot                                  18 Nov 16 16:00 UTC
                     |   16007dddd86b2ebb41d8880c53b77580cab13f02  18 Nov 16 11:23 UTC
                     |   a0eaf8c8ffc0b81c2199adaf7d67466891bb3205  17 Nov 16 10:22 UTC
                     |   1b664cc81d9cf25b213000686a9e8d6477ae1b06  16 Nov 16 20:31 UTC
                     |   ce1efe59c32d3c53344defdda4d5b643272022b8  16 Nov 16 20:29 UTC
                     |   4c33b216ae902ad89b9c3ca30a99a44833d16d5b  10 Nov 16 21:34 UTC
                     |   7015587783bc8f7a9c3d5cfa51e5a257fe188c4b  10 Nov 16 11:34 UTC
                     |   4d3ea2896c81e40a630d36cdc29ce740e7617016  10 Nov 16 11:30 UTC
                     :                                           
                     '-> 0.2.0                                     09 Nov 16 10:26 UTC
</code>

Flux helps you upgrade safely

Whoops, it is way behind! This brings us to the second big problem: upgrading a service can be error-prone. Our Kubernetes manifests (yamels) are kept in a Git repo, so that we always move from known state to known state. These are the steps we have to take when an image needs updating:

  1. Clone the repository
  2. Find the deployment files that use the image in question (there may be more than one, since we have e.g., sidecar containers that use the same image)
  3. Update a few fields in each file, in slightly different ways (e.g., put the image name in the template, and the tag part of the image name in a version label)
  4. Commit and push the change back to Git.
  5. Apply each deployment that was changed

Lots can go wrong, but even when it doesn’t, it’s dreadfully slow doing it all manually.

Flux can do all this mechanical work for you. If we just want to upgrade to the latest image, that can be calculated, too.

<code>
$ fluxctl release --service=sock-shop/front-end --update-all-images
Submitting release job...
Release job submitted, ID cc08f78c-0e66-24a9-9215-82c80cc646c3
Status: Complete.
Here's what happened:
1. Submitted job.
2. Calculating release actions.
3. Release latest images to sock-shop/front-end
4. Clone the config repo.
5. Clone OK.
6. Update 1 images(s) in the resource definition file for sock-shop/front-end: front-end (weaveworksdemos/front-end:0.2.0 -> weaveworksdemos/front-end:7f511af2d21fd601b86b3bed7baa6adfa9c8c669).
7. Update pod controller OK.
8. Commit and push the config repo.
9. Pushed commit: Release latest images to sock-shop/front-end
10. Regrade 1 service(s): sock-shop/front-end.
Took 55.810622454s
</code>

Now we have joined up all the stages: we can see the difference between what is running in the cluster and what could be running, and we can take action to reconcile them, while keeping a record of the new state so it can be rolled back or restored later.

You can try Flux now

We built Flux for our own purposes, but we are making it available to other people, as open source and in Weave Cloud. Sign up for Weave Cloud to try it out; or, take a look at the Flux Github repository. You can take a test drive or follow our detailed guide to learn how to integrate Flux and Weave Cloud into your own development environment.


Related posts

Empowering Platform & Application Teams: A Closer Look at Weave GitOps Enterprise Features

Kubernetes Security - A Complete Guide to Securing Your Containers

Multi-cluster Application Deployment Made Easy with GitOpsSets