Continuous Delivery with Weave Flux
Weave Cloud runs on Kubernetes. Kubernetes give us a suite of tools for managing a running system, but it does have boundaries. In particular, it doesn’t care (and shouldn’t care) where your container images come from. But we do — we have...
Weave Cloud runs on Kubernetes. Kubernetes give us a suite of tools for managing a running system, but it does have boundaries. In particular, it doesn’t care (and shouldn’t care) where your container images come from. But we do — we have taken great care to build a workflow around code review and continuous integration (CI).
Ultimately we want to go from code, to a tested container image, to a running service. Continuous integration (CircleCI and a bit of scripting, in our case) gets us through those first two steps, to a published container image. How do we make the final step?
We built Flux to link CI with Continuous Deployment (CD)
There are multiple sources of information about what is or should or could be running in your Kubernetes cluster. First, there is Kubernetes itself, which can tell us what is running. Second, there is the configuration, in the form of Kubernetes manifests, which we keep in git. Third, there is the container image registry, e.g., quay.io or Docker Hub, which is usually at the far end of your CI pipeline.
Flux tells you when there’s an upgrade ready
In practice, the problem is in two parts: lack of visibility and error-prone updates.
First of all, how do I tell what is running in my cluster and whether there’s a new image to which to upgrade?
We can see which services are running in Kubernetes, and we can see which images a deployment uses. In Weave Cloud we use the convention that a service has a single deployment associated with it, so combining those, we can see which images are running for a particular service. Using Flux:
<code> $ fluxctl list-services --namespace=sock-shop SERVICE CONTAINER IMAGE RELEASE POLICY sock-shop/cart cart weaveworksdemos/cart:0.3.0 sock-shop/cart-db cart-db mongo sock-shop/catalogue catalogue weaveworksdemos/catalogue:0.2.0 sock-shop/catalogue-db catalogue-db weaveworksdemos/catalogue-db:0.2.0 sock-shop/front-end front-end weaveworksdemos/front-end:0.2.0 sock-shop/orders orders weaveworksdemos/orders:0.3.0 sock-shop/orders-db orders-db mongo sock-shop/payment payment weaveworksdemos/payment:0.3.0 sock-shop/queue-master queue-master weaveworksdemos/queue-master:0.3.0 sock-shop/rabbitmq rabbitmq rabbitmq:3 sock-shop/shipping shipping weaveworksdemos/shipping:0.3.0 sock-shop/user user weaveworksdemos/user:0.3.0 sock-shop/user-db user-db weaveworksdemos/user-db:0.3.0 </code>
We can also see which images are available for a particular service using Flux.
<code>
$ fluxctl list-images --service=sock-shop/front-end
SERVICE CONTAINER IMAGE CREATED
sock-shop/front-end front-end weaveworksdemos/front-end
| 7f511af2d21fd601b86b3bed7baa6adfa9c8c669 18 Nov 16 16:00 UTC
| latest 18 Nov 16 16:00 UTC
| snapshot 18 Nov 16 16:00 UTC
| 16007dddd86b2ebb41d8880c53b77580cab13f02 18 Nov 16 11:23 UTC
| a0eaf8c8ffc0b81c2199adaf7d67466891bb3205 17 Nov 16 10:22 UTC
| 1b664cc81d9cf25b213000686a9e8d6477ae1b06 16 Nov 16 20:31 UTC
| ce1efe59c32d3c53344defdda4d5b643272022b8 16 Nov 16 20:29 UTC
| 4c33b216ae902ad89b9c3ca30a99a44833d16d5b 10 Nov 16 21:34 UTC
| 7015587783bc8f7a9c3d5cfa51e5a257fe188c4b 10 Nov 16 11:34 UTC
| 4d3ea2896c81e40a630d36cdc29ce740e7617016 10 Nov 16 11:30 UTC
:
'-> 0.2.0 09 Nov 16 10:26 UTC
</code>
Flux helps you upgrade safely
Whoops, it is way behind! This brings us to the second big problem: upgrading a service can be error-prone. Our Kubernetes manifests (yamels) are kept in a Git repo, so that we always move from known state to known state. These are the steps we have to take when an image needs updating:
- Clone the repository
- Find the deployment files that use the image in question (there may be more than one, since we have e.g., sidecar containers that use the same image)
- Update a few fields in each file, in slightly different ways (e.g., put the image name in the template, and the tag part of the image name in a version label)
- Commit and push the change back to Git.
- Apply each deployment that was changed
Lots can go wrong, but even when it doesn’t, it’s dreadfully slow doing it all manually.
Flux can do all this mechanical work for you. If we just want to upgrade to the latest image, that can be calculated, too.
<code> $ fluxctl release --service=sock-shop/front-end --update-all-images Submitting release job... Release job submitted, ID cc08f78c-0e66-24a9-9215-82c80cc646c3 Status: Complete. Here's what happened: 1. Submitted job. 2. Calculating release actions. 3. Release latest images to sock-shop/front-end 4. Clone the config repo. 5. Clone OK. 6. Update 1 images(s) in the resource definition file for sock-shop/front-end: front-end (weaveworksdemos/front-end:0.2.0 -> weaveworksdemos/front-end:7f511af2d21fd601b86b3bed7baa6adfa9c8c669). 7. Update pod controller OK. 8. Commit and push the config repo. 9. Pushed commit: Release latest images to sock-shop/front-end 10. Regrade 1 service(s): sock-shop/front-end. Took 55.810622454s </code>
Now we have joined up all the stages: we can see the difference between what is running in the cluster and what could be running, and we can take action to reconcile them, while keeping a record of the new state so it can be rolled back or restored later.
You can try Flux now
We built Flux for our own purposes, but we are making it available to other people, as open source and in Weave Cloud. Sign up for Weave Cloud to try it out; or, take a look at the Flux Github repository. You can take a test drive or follow our detailed guide to learn how to integrate Flux and Weave Cloud into your own development environment.
