This tutorial shows you how to configure a CI/CD pipeline using Google’s Cloud Platform (GCP) and Weave Cloud. The pipeline consists of automated build, test, packaging and deployment stages taking code through from commit to deployment into the Kubernetes cluster. When it’s up and running you’ll be able to commit new source code to your application and it will immediately deploy it into the cluster - the fastest way from code to a production deployment!
For the different stages of the pipeline we’ll use:
- Code repository: Google Cloud Platform Source Repositories (GSR)
- Continuous integration: Google Cloud Builder (GCB)
- Container registry: Google Container Registry (GCR)
- Continuous deployment: Weave Cloud Deploy
- Orchestrator: Google Kubernetes Engine (GKE)
Here’s a visualisation of the deployment pipeline:
All application code and configuration will be stored in a git repository. A commit merged into master will trigger the pipeline, causing a new container image to be built and deployed. We call this way of working ‘GitOps’ as it makes operating Kubernetes simple, repeatable and reversible.
You’ll need a Google Cloud Platform account which you can sign up for. Go into the GCP console and create a new project. For this tutorial we’ll use “GCP Weave CICD” with a project ID of “gcp-weave-cicd”.
Create a Code Repository
The start of the pipeline is a code repository for our application code. We’ll store our source code using GCSR.
Create a new repo by selecting “Source Repositories” then “Repositories” from the left-hand side menu. Click the link at the top of the page called “Create Repository” and create a repo called go-hello-world. The next page gives you three options:
Select the middle option: “Clone your Cloud Repository to a local Git repository”, then follow these steps:
1. Install Google Cloud SDK
Follow the instructions for your platform to OS to install the Google Cloud SDK.
2. Open a terminal and authenticate with GCP
Set the project as gcp-weave-cicd or whatever you’ve named it.
gcloud init
3. Clone your empty Cloud Repository to a local Git repository
gcloud source repos clone go-hello-world --project=gcp-weave-cicd
Note: This may display a ‘Warning: Remote HEAD refers to nonexistent ref. Unable to check out’ message that is safe to ignore.
4. Switch into your new local Git repository
cd go-hello-world
Great! You’ve now created a local git repository that’s linked to your Google Cloud Source Repository.
Building Hello World with GCB
The next step is to add some code, for this tutorial we’re going to create a simple “Hello World” HTTP server to demonstrate the pipeline. We’ll create a Dockerfile and give Google Cloud Builder (GCB) instructions to build the code into a Docker image. The steps are:
1. Create a ‘src’ directory
mkdir src
2. Create a file “src/hello-world.go” with the following contents
package main
import (
"fmt"
"log"
"net/http"
"os"
)
func main() {
port := "8080"
if fromEnv := os.Getenv("port"); fromEnv != "" {
port = fromEnv
}
server := http.NewServeMux()
server.HandleFunc("/", hello)
log.Fatal(http.ListenAndServe("0.0.0.0:"+port, server))
log.Printf("Listening on 0.0.0.0:%s", port)
}
func hello(w http.ResponseWriter, r *http.Request) {
log.Printf("Serving request: %s", r.URL.Path)
host, _ := os.Hostname()
fmt.Fprintf(w, "Hello world!\n")
fmt.Fprintf(w, "Hostname: %s\n", host)
}
3. Create a Dockerfile in the root of the repository
FROM alpine:3.5
COPY ./hello-world /hello-world
WORKDIR /
EXPOSE 8080
CMD [ "/hello-world", "-port", "8080"]
4. Tell GCB to build the Docker file
The convention is to call the GCB file “cloudbuild.yaml”:
steps:
- name: 'gcr.io/cloud-builders/go'
args: ['build', '-o', 'hello-world', 'src/hello-world.go']
- name: 'gcr.io/cloud-builders/docker'
args: ['build',
'-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$BRANCH_NAME-$SHORT_SHA',
'-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:latest',
'-f', 'Dockerfile', '.']
images:
- 'gcr.io/$PROJECT_ID/$REPO_NAME:$BRANCH_NAME-$SHORT_SHA'
- 'gcr.io/$PROJECT_ID/$REPO_NAME:latest'
As you can see there are two build steps; the first compiles the Go source file, the second builds the Dockerfile.
A nice feature of GCB is the ability to use environment variables in the build request file. Whatever you chose for your project and repo name, the file above will work. In our example the project was called ‘GCP Weave CICD’, so the build will result in a container image being created in Google Container Registry, with a path that looks something like this: “gcr.io/gcp-weave-cicd/go-hello-world:master-9709e0e”.
5. Commit and push everything
Commit your local changes and push to the remote GCSR repo:
git stage --all
git commit --message "initial commit"
git push --set-upstream origin master
You’ll end up with a repository that contains the following:
Create a Build Trigger
To automatically trigger building the source every time there’s a new commit we use a build trigger.
- In Google Cloud, select Cloud Build -> Triggers from the main menu.
- Click Create trigger in the main content area
- Select Cloud Source Repository as the source
- Select
go-hello-world(the git repository with the code) - In the trigger settings go to the Build configuration section and select
cloudbuild.yaml. Note that you may need to specify the precise location of the /cloudbuild.yaml file. - Create trigger to complete the wizard
Test that the build and trigger work by clicking Run trigger. The results will be put into Build history → Images pages, from the left-hand side navigation. You should have pair of image tags, entitled latest and one similar to master-7c6ce76:
Configure Kubernetes Manifests
The next stage is to create a configuration file (a manifest) that tells Kubernetes how to deploy the application. Every time we deploy a new version of the code we’ll update these manifests. We’ll store these operational configuration files in separate Git repo, this time on GitHub.
1. Create a repo in Github called go-hello-world-config.
Select the box Initialize this repository with a README.
2. Clone it to a local git repository:
git clone <github repo url> go-hello-world-config
3. Switch to your new local Git repository:
Create a directory called manifestsin the repo. We’ll add a couple of files to this directory that describe how the application should be run on the cluster.
cd go-hello-world-config
mkdir manifests
4. Create a file called manifests/hello-world-dep.yaml.
The first file describes how the application should be deployed, called a ‘Deployment’ in Kubernetes terminology. The contents of the file are:
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
annotations:
flux.weave.works/automated: "true"
name: hello-world
labels:
name: hello-world
spec:
replicas: 1
template:
metadata:
labels:
name: hello-world
spec:
containers:
- name: hello-world
image: gcr.io/gcp-weave-cicd/go-hello-world:master-9709e0e
ports:
- containerPort: 8080
Edit the image line for the container so that it has your project name, repo name and the tag that we created using the build trigger above. To find this go to GCP’s Container Registry, in the Build history section each build shows Artifacts. Cut and paste your specific configuration into the image line of your hello-world-dep.yaml file.
Note: If you edit image line incorrectly the deployment will fail.
5. Create a Service manifest
Next, create a Service, which tells Kubernetes how the application should be exposed to the world. Create a file manifests/hello-world-svc.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
labels:
name: hello-world
spec:
type: LoadBalancer
ports:
- port: 8080
targetPort: 8080
selector:
name: hello-world
This will tell Kubernetes to expose the application named ‘hello-world’, created by the ‘Deployment’ above, on port 8080 of an IP assigned to the cluster.
Running the application on GKE
If you don’t have one already, create a cluster in Google Kubernetes Engine (GKE). Follow the wizard to create a 3 node cluster.
In your terminal window connect to the cluster, the command is in the Connect button for the cluster. You only need the first command at this point.
Your local machine is now connected the cluster. We can deploy the application, using the manifests we created above. In the go-hello-world-config git repository root, run the following command:
kubectl apply -f ./manifests/
You should see output like this:
kubectl apply -f ./manifests/
deployment "hello-world" configured
service "hello-world" configured
To check the status of the deployment do:
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hello-world 1 1 1 1 2d
To check the service status do:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-world LoadBalancer 10.11.253.202 35.202.175.232 8080:32316/TCP 2d
kubernetes ClusterIP 10.11.240.1 <none> 443/TCP 2d
The application is quite simple, so it should’nt take long to be ready. We can use curl to check it, using the EXTERNAL-IP from the output above. The application is running on port 8080:
$ curl http://35.202.175.232:8080/
Hello world!
Hostname: hello-world-256096363-jq5d5
The final step is to push the manifests up to the GitHub repo:
git stage --all
git commit --message "Initial go-hello-world manifests"
git push
Set-up Weave Cloud
If you haven’t installed Weave Cloud through the Google Launcher yet, do that now. See Weave Cloud on Google Container Engine for the individual steps.
Configure Continuous Deployment
For Weave Cloud to automatically deploy new versions of the application it needs to know where the Kubernetes manifests are stored. Each time a deployment takes place it will update the manifest with the new container image tag before applying them to the cluster. In Weave Cloud click the cog icon on the top bar to go to the Configuration area. Under Config click on the Deploy menu.
There are two stages to the set-up:
- Give Weave Cloud access to the github repository
- Allow the agent on the cluster to deploy the latest version
GitHub configuration
For Weave Cloud to access and write to the Kuberenetes maniftests we tell it where the git repository is on GitHub.
Switch tabs to the GitHub page for your go-hello-world-config repository and select the Clone or download button. Make sure it’s showing Clone with SSH and then copy the URL for the repository.
Switch back to the Weave Cloud tab and paste this into the URL box in the Config Repository section. Remove the .git bit at the end so that it’s a proper URL. Set the Path to config files to manifests which is the directory we stored the manifests in. It should set Branch to be master for you automatically.
Scroll down the page a little bit and click on the PUSH KEY button.
This will take you to a page on Github that asks you to Authorize Weave Cloud. These permissions let Weave Cloud read and write the Kubernetes manifests from Github. After accepting these you’ll be taken back to Weave Cloud.
Configure the Agent
Next configure the agents on the cluster. Click the cog icon on the main bar to go to the configuration page. Take the Config → Deploy menu.
Re-fill out the Config Repository section with the URL and the Path to config files. As you do this it will update the code snippet just below. Copy the code snippet from the page (use the Copy button on the right) and run it in your terminal:
$ kubectl apply -f "https://cloud.weave.works/k8s/flux.yaml?t=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa&flux-version=%5E1&k8s-version=$(kubectl version | base64 | tr -d '\n')&git-label=flux-muddy-fire-55&git-url=git%40github.com%3Auser_name%2Fgo-hello-world&git-path=manifests&git-branch=master"
serviceaccount "weave-flux" configured
clusterrole "weave-flux" configured
clusterrolebinding "weave-flux" configured
secret "flux-git-deploy" configured
deployment "weave-flux-memcached" configured
service "weave-flux-memcached" configured
deployment "weave-flux-agent" configured
Return to the Weave Cloud Config page and scroll down and click on the PUSH KEY button. To test that it’s correctly configured you can take the link next to the button which will take you to the Deploy keys section of Github in a new tab.
We’ve now told the agents on the cluster how to access the repository and set-up a deployment key. As the agents reconfigure themselves the page will show a yellow warning notice. This can take a few moments, and then it will show a green notice with AGENTS CONFIGURED.
When you have the green notice click the Deploy button in the main navigation. You’ll now see a list of the services in the cluster. Select the one entitled default:deployment/hello-world and check that it says DEAUTOMATE. If the button says AUTOMATE click it to change it. When it’s set to automated the service name on the left will show a small truck icon:
By putting it into automate mode we’re telling Weave Cloud that for this service we want it to:
- Monitor the container image specified in the Deployment, for new tag versions
- Whenever a new tag is discovered, update the Deployment manifests
- Commit and push the change in git
- Apply the updated manifest to the Kubernetes cluster
Deploy a change, completely automatically!
The time has come to deploy a new version of the containerized “hello world” web server, just by committing a change to the application code.
1. Change the application source code
In your terminal change into the go-hello-world/src/ source repository. In hello-world.go change the output statement text from “Hello world!” to something new:
func hello(w http.ResponseWriter, r *http.Request) {
log.Printf("Serving request: %s", r.URL.Path)
host, _ := os.Hostname()
fmt.Fprintf(w, "Hello! The CICD pipeline works!\n")
fmt.Fprintf(w, "Hostname: %s\n", host)
}
2. Commit and kick off the pipeline
When we commit and push the new code to the source repository it will automatically kick off the pipeline.
$ git add src/hello-world.go
$ git commit -m 'change text'
[master 840c5ed] change text
1 file changed, 1 insertion(+), 1 deletion(-)
$ git push
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 354 bytes | 354.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
To https://source.developers.google.com/p/$PROJECT_NAME/r/go-hello-world
7c6ce76..840c5ed master -> master
3. Check each stage of the pipeline automation
We can check that each stage of the deployment pipeline has worked.
- Check the code is pushed by visiting Source Repositories on GCP. Go into Source Code to view the code repository.
- Check a build happened by going to Container Registry on GCP. Check in Build history for a new build.
- Also check a new container image was created in Container Registry. In “Images” go into the
go-hello-worldrepo where you should see a new container image with new tags.
4. Check Weave Cloud deployed the new version
Go into “Deploy” on Weave Cloud into the default:deployment/hello-world service. In the “Containers section on the page it tells you about the container that it deployed, something like gcr.io/gcp-weave-cicd/go-hello-world:master-cc97017 which should be the same tags as in GCP’s Container Registry.
In the History on the right it will say something like Synced to cluster: default:deployment/hello-world (4c5837b022ff) click on the git has to be taken across to the config repository on GitHub where you can see the change in the manifest.
5. Test the application
When the new image has been deployed, you can test the application:
$ curl http://35.202.175.232:8080/
Hello! The CICD pipeline works!
Hostname: hello-world-256096363-3dkxt
As you can see we get the new updated message! Note that the pod name has changed, that’s because Kubernetes deployed a new pod when the image was updated.
Summary
Congratulations, you now have a fully automated CI/CD pipeline - the fastest way from commit to production deployment! The pipeline joins together Google Source Repositories, Google Cloud Builder, Google Container Registry and Weave Cloud. To take this further add your own application and build out more complex deployments through the pipeline.
Weave Cloud joins together continuous deployment, observability and metrics monitoring. As you deploy more regularly you’ll want to partner that with monitoring your applications for performance and reliability. To learn more take these links:
- Learn about different aspects of Weave Cloud in our Tutorials
- Read about metrics monitoring using the RED method