Guide to Installing Spinnaker in Kubernetes Clusters
Learn how to have a production-grade Spinnaker which can be used to create your deployment pipelines for one or more Kubernetes clusters.
Spinnaker is an open-source, multi-cloud deployment tool for releasing software changes reliably. You can deploy and automate your application release across multiple cloud environments including Kubernetes, AWS EC2, Google Computer Engine, Google App Engine, Microsoft Azure and many more.
In this post, we will focus on the installation of Spinnaker in a kubernetes cluster using Halyard. Halyard is a command-line administration tool that manages the lifecycle of your Spinnaker deployment, and it's a recommended way to install, configure and update Spinnaker.
We will use Google Cloud for demonstration, the process would be the same for other cloud providers with slight modifications in commands and resources.
Prerequisites
- One Kubernetes cluster Spinnaker would be set-up in this cluster using Google Kubernetes Engine for the demonstration.
- One Virtual Machine (1 vCPU, 3.75 GB, Ubuntu 18.04) for installation of Halyard.
- One storage bucket for persistent storage of Spinnaker.
- One service account json key for access storage bucket.
- An OAuth client ID and access token for Login authentication.
We will create all these resources in subsequent steps. Kindly replace your-project with your actual project in required commands.
Steps For The Installation And Execution
Step 1: Create A Kubernetes Cluster
You can skip this step if you already have one. We will use a “one node” single-zone cluster for demonstration:
gcloud beta container clusters create "kubernetes-cluster-01" \
--zone "asia-southeast1-a" --machine-type "n1-standard-1" \
--num-nodes "1" --no-enable-stackdriver-kubernetes \
--project "your-project"Step 2: Create One Machine To Install Halyard.
gcloud compute instances create "spinnaker-halyard" \
--async \
--boot-disk-size "10GB" \
--image-family "ubuntu-1804-lts" \
--image-project "ubuntu-os-cloud" \
--machine-type "n1-standard-1" \
--zone "asia-southeast1-a" \
--project "your-project"Step 3: Log in To The Machine
gcloud beta compute ssh --zone "asia-southeast1-a" "spinnaker-halyard" --project "your-project"Step 4: Install Kubectl And Halyard
Install kubectl:
sudo -i
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" |
sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectlInstall Halyard:
Add spinnaker user and switch to the spinnaker user, then install Halyard:
adduser spinnaker
echo "spinnaker ALL=(ALL:ALL) NOPASSWD:ALL" > spinnaker
&& chmod 440 spinnaker && mv spinnaker /etc/sudoers.d/
su - spinnaker
curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install
/debian/InstallHalyard.sh
sudo bash InstallHalyard.sh --user spinnaker
(Press Enter and keep default option when it will prompt)
. /home/spinnaker/.bashrc
hal -vspinnaker@spinnaker-halyard:~$ hal -v
1.31.1-20200207105616We have successfully installed the Halyard after completion of this step.
Step 5: Create An IAM Service Account And It’s Key
Spinnaker will use this service account to access persistent GCS storage:
gcloud iam service-accounts create spinnaker \
--display-name "spinnaker" \
--project your-project
gcloud iam service-accounts keys create ~/.gcp/key.json \
--iam-account spinnaker@your-project.iam.gserviceaccount.com \
--project "your-project"Step 6: Create A Storage Bucket
If you want to persist data generated by Spinnaker, you will need to create a storage bucket with the proper access rights. You will need to give the right permissions on this bucket to the Spinnaker service account created in Step 5. Feel free to change BUCKET_NAME as storage buckets are Global resources and must be unique:
BUCKET_NAME=spin-hal-bucket
gsutil mb -l asia-southeast1 -p your-project gs://$BUCKET_NAME/
gsutil iam ch serviceAccount:spinnaker@your-project.iam.gserviceaccount.com:
legacyBucketWriter gs://$BUCKET_NAMEStep 7: Connect To Your Kubernetes Cluster On Your Halyard Machine
gcloud beta container clusters get-credentials "kubernetes-cluster-01"
--zone "asia-southeast1-a" --project "your-project"Step 8: Create Kubernetes Service Account
Create spinnaker-service-account and assign ClusterAdmin role to this service account.
CONTEXT=$(kubectl config current-context)
kubectl apply --context $CONTEXT -f https://spinnaker.io/downloads/kubernetes/service-account.yml
TOKEN=$(kubectl get secret --context $CONTEXT
$(kubectl get serviceaccount spinnaker-service-account
--context $CONTEXT -n spinnaker -o jsonpath='{.secrets[0].name}') -n spinnaker
-o jsonpath='{.data.token}' | base64 --decode)
kubectl config set-credentials ${CONTEXT}-token-user --token $TOKEN
kubectl config set-context $CONTEXT --user ${CONTEXT}-token-userStep 9: Add The Kubernetes Cluster Into Halyard Config
hal config provider kubernetes enable
hal config provider kubernetes account add kubernetes-cluster-01 --provider-version v2
--context $(kubectl config current-context)
hal config features edit --artifacts trueRepeat steps 7 to 9 for each of the clusters in-which you want to manage using Spinnaker or want to install Spinnaker in.
Step 10: Choose The Kubernetes Cluster in Which You Want To Install Spinnaker.
ACCOUNT=kubernetes-cluster-01
hal config deploy edit --type distributed --account-name $ACCOUNTStep 11: Add Storage Bucket in Halyard Config.
hal config storage gcs edit --bucket $BUCKET_NAME --project your-project
--json-path ~/.gcp/key.json
hal config storage edit --type gcsStep 12: Check Available Spinnaker Versions And Deploy
Now List the available spinnaker versions. Note that this will list the latest available version of Spinnaker and not Halyard:
hal version listspinnaker@spinnaker-halyard:~$ hal version list
+ Get current deployment
Success
+ Get Spinnaker version
Success
+ Get released versions
Success
+ You are on version "", and the following are available:
- 1.16.7 (SecretObsession):
Changelog: https://gist.github.com/spinnaker-release/170d178708b56e83b0289452cb83f347
Published: Mon Mar 09 19:34:31 UTC 2020
(Requires Halyard >= 1.17)
- 1.17.7 (LlamaLlama):
Changelog: https://gist.github.com/spinnaker-release/eea8c2434c8dcf77de8506ffec705246
Published: Mon Mar 09 19:37:42 UTC 2020
(Requires Halyard >= 1.17)
- 1.18.5 (Longmire):
Changelog: https://gist.github.com/spinnaker-release/2cfb72a46883b82a657b4e4b8cba6f62
Published: Mon Mar 09 19:44:00 UTC 2020
(Requires Halyard >= 1.29)
- 1.19.0 (Gilmore Girls A Year in the Life):
Changelog: https://gist.github.com/spinnaker-release/dbc44ac411d5076002b5db7c64b8c63e
Published: Wed Mar 11 20:22:58 UTC 2020
(Requires Halyard >= 1.32)Set VERSION with one of the listed versions:
VERSION=1.16.7
hal config version edit --version $VERSIONDeploy Spinnaker:
hal deploy applyAfter this step, Spinnaker would have been deployed in the spinnaker namespace of kubernetes cluster. Note that before hal deploy apply, each configuration was added in hal configuration only and would be applicable on running spinnaker after hal deploy apply command.
We can check all Spinnaker microservices running in spinnaker namespace:
kubectl get all -n spinnakerspinnaker@spinnaker-halyard:~$ kubectl get all -n spinnaker
NAME READY STATUS RESTARTS AGE
pod/spin-clouddriver-78f5977c8b-6wll2 1/1 Running 0 45m
pod/spin-deck-7d86cd5c56-bnjlt 1/1 Running 0 45m
pod/spin-echo-7d9dcb8894-ggln4 1/1 Running 0 45m
pod/spin-front50-78996dfbf4-g5g2m 1/1 Running 0 45m
pod/spin-gate-5dfd759658-rwxr5 1/1 Running 0 45m
pod/spin-orca-76d64868db-lb7cp 1/1 Running 0 45m
pod/spin-redis-c9b94cd9c-5lz5z 1/1 Running 0 45m
pod/spin-rosco-64544867b8-vzvz8 1/1 Running 0 45m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/spin-clouddriver ClusterIP 10.43.254.196 7002/TCP 10d
service/spin-deck ClusterIP 10.43.240.189 9000/TCP 10d
service/spin-echo ClusterIP 10.43.252.206 8089/TCP 10d
service/spin-front50 ClusterIP 10.43.250.115 8080/TCP 10d
service/spin-gate ClusterIP 10.43.248.203 8084/TCP 10d
service/spin-orca ClusterIP 10.43.247.203 8083/TCP 10d
service/spin-redis ClusterIP 10.43.242.44 6379/TCP 10d
service/spin-rosco ClusterIP 10.43.253.57 8087/TCP 10d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/spin-clouddriver 1/1 1 1 10d
deployment.apps/spin-deck 1/1 1 1 10d
deployment.apps/spin-echo 1/1 1 1 10d
deployment.apps/spin-front50 1/1 1 1 10d
deployment.apps/spin-gate 1/1 1 1 10d
deployment.apps/spin-orca 1/1 1 1 10d
deployment.apps/spin-redis 1/1 1 1 10d
deployment.apps/spin-rosco 1/1 1 1 10d
NAME DESIRED CURRENT READY AGE
replicaset.apps/spin-clouddriver-78f5977c8b 1 1 1 10d
replicaset.apps/spin-deck-7d86cd5c56 1 1 1 10d
replicaset.apps/spin-echo-7d9dcb8894 1 1 1 10d
replicaset.apps/spin-front50-78996dfbf4 1 1 1 10d
replicaset.apps/spin-gate-5dfd759658 1 1 1 10d
replicaset.apps/spin-orca-76d64868db 1 1 1 10d
replicaset.apps/spin-redis-c9b94cd9c 1 1 1 10d
replicaset.apps/spin-rosco-64544867b8 1 1 1 10dInfo About Spinnaker Microservices:
- Clouddriver: Responsible for integration of cloud providers like AWS, GCP, Azure etc.
- Deck: Management UI for Spinnaker.
- Gate: Spinnaker API Gateway.
- Echo: Spinnaker Eventing Service (sending notifications, acts on incoming webhooks etc).
- Front50: Spinnaker Metadata Repository Service to persist the metadata of pipelines, projects etc.
- Orca: Orchestration engine for Spinnaker.
- Rosco: Produces machine images for Spinnaker.
Step 13: Expose Spinnaker UI Outside Kubernetes Cluster
Deck is the browser-based UI for spinnaker management. We need to expose the deck outside the Kubernetes cluster to access this from our browser. We would achieve this by editing spin-deck service and making it a type of Internal load balancer.
kubectl edit svc spin-deck -n spinnakerAdd the following parameter in the spin-deck yaml file according to kubernetes convention.
In Metadata:
annotations:
cloud.google.com/load-balancer-type: InternalIn Spec:
- port: 80
type: LoadBalancerFinal yaml For Spin-deck After Changes:
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/load-balancer-type: Internal
labels:
app: spin
cluster: spin-deck
name: spin-deck
namespace: spinnaker
spec:
clusterIP: 10.43.240.189
externalTrafficPolicy: Cluster
ports:
- port: 80
protocol: TCP
targetPort: 9000
selector:
app: spin
cluster: spin-deck
sessionAffinity: None
type: LoadBalancerStep 14: Expose Spinnaker Gate
Gate is the API gateway for spinnaker. All api callers communicate with Spinnaker via Gate.
kubectl edit svc spin-gate -n spinnakerSimilar to Step 13 add following in spin-gate yaml
In Metadata:
annotations:
cloud.google.com/load-balancer-type: InternalIn Spec:
- port: 80
type: LoadBalancerFinal yaml For Spin-gate After Changes:
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/load-balancer-type: Internal
labels:
app: spin
cluster: spin-gate
name: spin-gate
namespace: spinnaker
spec:
clusterIP: 10.43.248.203
externalTrafficPolicy: Cluster
ports:
- port: 80
protocol: TCP
targetPort: 8084
selector:
app: spin
cluster: spin-gate
sessionAffinity: None
type: LoadBalancerWe can check if both deck and gate have the same type of Internal Load balancer service now.
kubectl get svc -n spinnakerspinnaker@spinnaker-halyard:~$ kubectl get svc -n spinnaker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/spin-clouddriver ClusterIP 10.43.254.196 7002/TCP 10d
service/spin-deck LoadBalancer 10.43.240.189 10.148.0.49 80:32261/TCP 10d
service/spin-echo ClusterIP 10.43.252.206 8089/TCP 10d
service/spin-front50 ClusterIP 10.43.250.115 8080/TCP 10d
service/spin-gate LoadBalancer 10.43.248.203 10.148.0.50 80:31320/TCP 10d
service/spin-orca ClusterIP 10.43.247.203 8083/TCP 10d
service/spin-redis ClusterIP 10.43.242.44 6379/TCP 10d
service/spin-rosco ClusterIP 10.43.253.57 8087/TCP 10dStep 15: Add Domains in Halyard Config And Deploy Spinnaker
Assign domains on both decks’ and gates’ Load balancer IPs, then add these domains in Halyard config. Let’s use the following domains for the rest of the tutorial, replace these domains with your actual domains in respective commands.
spin-deck -> spinnaker-ui.example.com
spin-gate -> spinnaker-api.example.comAdd these domains in halyard config and deploy them again:
hal config security ui edit --override-base-url http://spinnaker-ui.example.com
hal config security api edit --override-base-url http://spinnaker-api.example.com
hal deploy applyAfter this step you would be able to access Spinnaker using http://spinnaker-ui.example.co... domain.
Step 16: Add Login Authentication On Spinnaker
We will add Google OAuth for Login into Spinnaker and will allow Login for particular domains only. We would need Google OAuth client Id and secret for this setup. To generate them, follow the steps below:
- First navigate to https://console.cloud.google.c...
- Click on "CREATE CREDENTIALS" and then on "OAuth client ID"
- Select "Web application" in Application type, and enter "Spinnaker" in the Name text box. After that under "Authorized redirect URIs", add http://spinnaker-ui.example.co.../login and http://spinnaker-api.example.c... (Replace these domains with your actual domains created in Step 15). After that click on Create and Copy Client ID and Client Secret.
Step 17: Enable OAuth in Spinnaker
CLIENT_ID=CLIENT_ID_GENERATED_IN_STEP_16
CLIENT_SECRET=CLIENT_SECRET_GENERATED_IN_STEP_16
PROVIDER=google
DOMAIN=example.com
hal config security authn oauth2 edit --client-id $CLIENT_ID --client-secret $CLIENT_SECRET
--provider $PROVIDER --user-info-requirements hd=$DOMAIN
hal config security authn oauth2 enable
hal deploy applyAfter this step, We have successfully set up Authentication for our Spinnaker UI. We can open our Spinnaker UI Domain at http://spinnaker-ui.example.co... and can Login with our example.com Email Id.
TL;DR
- Weave Gitops helps release applications with high velocity and reliability.
- We used Halyard (a command-line administration tool) to install Spinnaker in Google Kubernetes Engine.
- We exposed Spinnaker UI and API Gateway within the network using Internal Load Balancer and added Login authentication.
- We can add more functionality like slack/email notification, Automatic pipeline trigger from jenkins, Role based authorization etc. which we will try to cover in separate threads.
Whitepaper: Production Ready Checklists for Kubernetes
Download these comprehensive checklists to help determine your internal readiness and gain an understanding of the areas you should update
