GitOps With GitHub Actions and EKS
Curious about GitHub Actions and GitOps? This step by step tutorial shows you how to use GitOps and GitHub Actions for application deployments to EKS. Learn how to set up an EKS cluster with eksctl, GitHub Actions to push the image to ECR, then deploy and commit that image tag back to GitHub with Weave Flux.
Guest post by Jeremy Adams - @jpadamspdx and John Bohannon - @imjohnbo
Hi! We’re Jeremy Adams and John Bohannon from the GitHub Business Development team. Like the best haircuts, our team is a bit of business in the front (like Jeremy) and party, err engineering, in the back (like John). In reality our engineering teammates all enjoy working closely with our ecosystem partners on strategic projects. Many of us “business types” also like getting our hands on keyboard, which makes working in BD a blast.
When we started our latest project, our goal was simple to state: A Weave GitOps workflow with GitHub Actions and Amazon EKS.
Now where to start? When you want to deeply understand how various tools fit together into a more complex chain or system, you can just read about it, but we prefer to actually build it; starting with a simple working sketch of a system that can be extended into something more real-world. The goal is to ultimately create something that adds value to your organization or to the community at large, but friction and unnecessary complexity at this early stage are far from helpful.
In that spirit, we present this end-to-end technical walkthrough where you’ll get the hands-on experience of:
- setting up an EKS cluster with Weaveworks eksctl
- using a GitHub Actions workflow to build and push a new container image to AWS ECR upon code change
- deployment to EKS and commit of that new image tag back to GitHub with Weave Flux
Basically something like this:
Step 1: Setup AWS user, initial CLI tools, ECR
Okay, we’ve got an outline and a diagram, but we’re going to need to get some tools that work together to make this happen. You’ll need GitHub and AWS accounts for this exercise. We’re going to assume you have basic proficiency in git, Linux shell, basic container/Kubernetes concepts, and can get an AWS account, and IAM policy yourself or through your friendly neighborhood IT admin.
There are great resources for ekstctl from Weaveworks and AWS, but I’ll describe my exact process so you can follow along.
- Add a user with the perms needed for eksctl and Flux to create an EKS cluster and push/pull the image in ECR. We created a new IAM Policy called “eksctl-policy” and used this IAM Policy JSON with our AWS Account ID filled in.
- Create a new IAM user with Programmatic Access. For permissions choose “Attach existing policies directly”. We then attached two permissions in order to create/destroy EKS clusters with eksctl and to create ECR repos and push/pull :
- Our custom “eksctl-policy”
- The AWS-managed “AmazonEC2ContainerRegistryPowerUser” policy - Check that the aws CLI, eksctl, and fluxctl are installed locally to your workstation. Collect your user credential information and run
aws configureto set your credentials and the AWS region.
NOTE: Ensure that you have sufficient Elastic IPs and VPCs in the AWS region you select (five by default per region). I decided to simply change to a region that had EKS available and that wasn’t being used heavily by my organization. - Create an Amazon Elastic Container Registry (ECR) repository called “example-eks” in the same AWS region you’ll create the EKS cluster. I used the AWS console to create the repo. The ECR is where we’ll push our container images to and for Weave Flux (more about Flux in a moment) to deploy them from.
Step 2: Create the EKS cluster
Now that we have a user with the right permissions, our CLI tools, and an ECR repo. Let’s create the EKS cluster! We love to use eksctl for this, because it handles all of the tricky details of setup (and you can run eksctl delete cluster when you’re done). There are lots of eksctl options, but it can be as simple as:
$ eksctl create cluster
It will take about 10-20 minutes to get your new cluster.
...
[✔] saved kubeconfig as "/Users//.kube/config"
...
[✔] EKS cluster "improbable-name-9999999999" in "eu-west-1" region is ready
Since there is a new kubeconfig in place, when you run kubectl from your desktop, you’ll be interacting with your new EKS cluster! The generated cluster name is in the last line of output, but if you forget, just run one of these commands.$ eksctl get clusters$ aws eks list-clusters
Step 3: Setup our GitHub Actions Workflow repo
We’re going to use GitHub Actions to drive our container build CI from a workflow file in the GitHub repo.
To set up yours:
- Fork the example repo into your GitHub user account. https://github.com/github-developer/example-actions-flux-eks
- Navigate to the .github directory in the GitHub web interface Code tab. Click on the Actions tab and enable GitHub Actions workflow in this forked repo.
3. View the workflow in your repo under .github/workflows/build.yml. We’ll walk through the whole workflow, but let’s make sure you have it set up first, starting with the environment variables on lines 16-20.
16 AWS_DEFAULT_REGION: eu-west-1 17 AWS_DEFAULT_OUTPUT: json 18 AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}19 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}20 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
Ensure that line 16 has the region where your EKS cluster resides. This should match the region in your ~/.aws/config or the region specified on the eksctl command line.
Step 4: Configure secrets
Lines 18-20 in the build.yml file refer to secrets. Secrets are set up in the Settings tab, which are found at the bottom of the left navigation menu in the Secrets section.
Create three secrets called AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY with your AWS account ID (obtainable from the AWS console or by running aws sts get-caller-identity)and the new user’s key ID and the secret key (found in your ~/.aws/credentials file). Secrets associated with your GitHub repo allow you to keep sensitive data out of your source code, but still make them available to GitHub Actions workflows.
Step 5: App development workflow
Now is a good time to take a look around the repo. Notice that the simple containerized web app we’re going to deploy is in the app directory, whereas the Kubernetes YAML files for the Deployment and Service are in the manifests directory. The development workflow that we’re going to employ is as follows:
- Developers push changes to the app directory.
- GitHub Actions then builds and pushes container images to Elastic Container Repository (ECR), as specified by a workflow, .github/workflows/build.yml.
- Flux notices a new image in ECR and deploys it to EKS by modifying the deployment manifest, and memorializing that change with a
git commitback to manifests/deployment.yml.
Speaking of ECR, remember the ECR repo we created called “example-eks”? That repo name will match up with the name used for the image repo:tag when we build it with the GitHub Actions workflow.
Now we’re going to get GitHub Actions to push our first container image there!
Step 6: Priming the pump for our first deployment
Before turning over the continuous deployment duties of new container images to Flux, we need to push at least one successful initial deployment. We’ll make a change to our app that triggers our GitHub Actions workflow to build a new container image, tag it with the current git SHA, and push it to ECR. That image will then be used in our deployment.yml as a starting point, and Flux can take it from there.
Commit a change and copy the image URI
To kick off that build let’s make a simple CSS change that switches the text color from white on a black background to green on black. It’s easy to do this from the GitHub web interface, and can also be done on the command line.
Once you commit the change, switch over to the Actions tab to view the running workflow:
1. Click on the workflow to view the steps.
2. Expand the Build and tag the image step, to view most of the new image repo and the tag on the last line.
3. Take a look at your “example-eks” repo in ECR and copy the full image URI.
4. Update manifests/deployment.yml with the image URI you just copied from the AWS console.
Now you’re ready to install Flux on the EKS cluster and have it perform its first sync!
But first, a quick check-in. Almost there!
Let’s take stock of where we are:
- We have our CLI tools, an EKS cluster, and an ECR repo.
- GitHub Actions workflow is pointed at the correct region and has the credentials it needs to push a new image to ECR in its secrets. The manifests/deployment.yml has the correct ECR repo noted as well (we just did that a moment ago).
- Weave Flux to watch for when GitHub Actions workflow builds and pushes a new container image to the ECR repo.
Okay, we’re ready. Let’s get Flux running on our cluster, and synchronize things for the first time. Then we’ll make a change to our app to get a new container image, and watch the magic happen.
Step 7: Run Weave Flux on the EKS cluster
We’re going to run Flux on our EKS cluster in its own “flux” namespace. We’ll use this short script (also in the README.md) to make it happen. Be sure to run it from the root of your example-actions-flux-eks repo. Notice that flux is watching the manifests folder for deployment YAML changes.
export GHUSER=
export GHREPO=example-actions-flux-eks
kubectl create ns flux
fluxctl install \
--git-user=${GHUSER} \
--git-email=${GHUSER}@users.noreply.github.com \
--git-url=git@github.com:${GHUSER}/${GHREPO} \
--git-path=manifests \
--namespace=flux | kubectl apply -f -
If all went well, you should have Flux running on your cluster.
serviceaccount/flux created clusterrole.rbac.authorization.k8s.io/flux unchanged clusterrolebinding.rbac.authorization.k8s.io/flux unchanged deployment.apps/flux created secret/flux-git-deploy created deployment.apps/memcached created service/memcached created
Check it by running:
$ kubectl get all -n flux
Since we want Flux to be able to write back to our git repo the tag of the image it deploys, we need to give it read/write access to our repo with a Deploy Key. Run this command to get the Flux public key and add it.
$ fluxctl identity --k8s-fwd-ns flux
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC64WoWesnPneyDqq8ddTAAOKSaLHcu+0ALL8xxtGdnbK2WG99OZ7A9cq24Y9TmSL4gIuXb0HDvwhHsnbkTNsFmVWpO9xS/T3bqhLzhdQwLCGP21ckhRVF7RBv+pK6PnenY4ZjTRkW5h7SxYnunEarj/9E9NlL/JP8tDnb53liDXF4AB1y3Xi/nKwjlgwkGGrSBXGSRij7a6uq2iMlGF/H9MmHn8ct7w/dd/RF6VN4phbNpsVfnBVu1yDgRJTNKznXDOCEEAfflxAFrDWjbAsXwCxvWLNsbP5HtMTf5Ep/Eba7ZAjZ7XnWYLgoXRZHOf+0WYqn1EfsSot5pb01TFeYr
NOTE: Be sure to check the box to allow write access!
Step 8: Sync the deployment and view the app!
To get Flux and GitHub in sync for this first go-round, issue a manual Flux sync command. Flux immediately ensures that the state of objects in Kubernetes for which we have a YAML file match.
$ fluxctl sync --k8s-fwd-ns flux Synchronizing with git@github.com:jpadams/example-actions-flux-eks Revision of master to apply is 1fbb995 Waiting for 1fbb995 to be applied ... Done.
Let’s find out where we can view our app running in the default Kubernetes namespace and browse to the DNS name of the ELB.
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE aws-example-octodex LoadBalancer 10.100.64.193 80:31273/TCP 20h
You should be seeing an Octocat in green, spouting wisdom. Hurray!
After this, we’re going on a sort of auto-pilot. Whenever we change our app code, GH Actions will build a new container image and push it to ECR. Flux will notice the new image, deploy it to the cluster, and then sync up the actual Kubernetes Deployment with the manifests/deployment.yml. From here on out, the state of the cluster will win. Let’s try that out.
Step 9: Make a change to the app and push to master
We’ll make a change to our app code again and push it to master, but this time we won’t do anything else but wait (for about five or six minutes).
Let’s change the CSS again to go from green to yellow this time.
Feel free to peek at the Actions tab and the AWS ECR console to see how things are going with the build and push of the new image. While we wait a few minutes for Flux to deploy the change, let’s read a little more about the workflow.
More details on the GitHub Actions workflow
You might have noticed that in the .github/workflows/build.yml we configured the workflow to only run when there’s either a PR or a push to master that involves files under the app directory. Changes anywhere else in the repo’s file tree don’t trigger a workflow run. That’s a good thing because we don’t want Flux’s writes to the manifests directory to trigger another run post deployment (we could have also ignored that directory path).
on:
pull_request:
paths:
- app/**
push:
paths:
- app/**
branches: # array of glob patterns matching against refs/heads. Optional; defaults to all
- master # triggers on pushes that contain changes in master
Building and tagging images
The middle of the workflow involves building and tagging (but not pushing quite yet) the container image to ECR. This will run for every PR or push to master. There are comments suggesting where you might want to insert linting, unit tests, or security scans. All good ideas, but beyond the scope of our basic example.
name: Build and Push to ECR
env:
AWS_DEFAULT_REGION: eu-west-1
AWS_DEFAULT_OUTPUT: json
AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
CONTAINER_IMAGE: example-eks:${{ github.sha }}
jobs:
build-and-push:
name: Build and deploy
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@master
# Add steps here like linting, testing, minification, etc.
- name: Setup ECR
run: |
# Login to AWS ECR
$( aws ecr get-login --no-include-email )
- name: Build and tag the image
run: |
# Build and tag the image
docker build \
-t $CONTAINER_IMAGE \
-t $GITHUB_REPOSITORY:$GITHUB_SHA \
-t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE ./app
# Add additional steps here like scanning of image
Pushing images to the container repo
The last step in the workflow is where the push to ECR occurs, but we’ve added an if condition to ensure that we only run that step if it’s a push to master, and not just a PR.
# Only push to registry on master
- name: Push
if: github.ref == 'refs/heads/master'
run: |
# Push image to AWS ECR
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE
After five minutes are up, refresh the app web page or check if a new pod has started in Kubernetes.
$ kubectl get pods NAME READY STATUS RESTARTS AGE aws-example-octodex-5b87d5f56f-wdk8d 1/1 Running 0 62s
Flux, the GitOps operator
Great that worked, but clearly I’m running a new container image with the “yellow” version. The last time I touched my deployment.yml, I had specified the old “green” version of my app’s container image. Better take a look!
Woot! As promised, Weave Flux not only noticed and deployed the new container image associated with the new app version, it also made a commit in my repo to record the change. Plus notice that the container image tag matches the hash of the commit that generated it! That’s what I call GitOps!
Conclusion
You have just found out first-hand how to set up a real deal GitOps workflow with GitHub Actions and Weaveworks tools on EKS. Or maybe you browsed and skimmed and got a sense of what’s possible. Either way, thanks for reading!
From here you could explore more advanced patterns of CI or deployment with GitHub Actions like adding linting, security scanning, or using exploring the GitHub Actions Marketplace for more CI/CD building blocks.
Or you could move into more advanced Kubernetes and Flux usage with Helm charts and Flux HelmReleases, or push the state of the art with a service mesh (e.g. AWS AppMesh, Istio, or Linkerd) and Weaveworks Flagger for Progressive Deployment (PD).
We can’t wait to see what you build and share!
Have questions on what you need to create a cloud native platform?
The Weaveworks team can help you navigate the vast landscape of cloud native technologies – OSS and managed on your behalf. Together we can create a cloud native reference architecture that fits your business needs. You can benefit from a Weaveworks’ validated design or you can design, review and select technology options with our expertise.
Contact us for a demo of Weave GitOps Enterprise, jump right in and get GitOps set up with two commands with our free Weave GitOps Core product, or check out our Quickstart program to help you get up and running with Kubernetes.
