This is Part 4 of 4 of the Weave Cloud core tutorial series.
In this guide, how to secure your app by defining Kubernetes Network Policy and having it enforced by Weave Net is demonstrated.
Securing segments of your app is simple with the application of Kubernetes-based policy that is enforced by Weave Net. Add namespaces to the policy yaml files to create software firewalls and visualize the result in Weave Cloud.
A Video Overview
Sign up for a Weave Cloud account
Go to Weave Cloud and register for an account. You’ll use the Weave Cloud token later to send metrics to Cortex and view Weave Net with it.
Deploy a Kubernetes Cluster with Weave Net and the Sample App
If you have already done this as part of one of the other tutorials, you can skip this step. Otherwise, click “Details” below to see the instructions.
This example uses Digital Ocean, but you can just as easily create three instances in AWS, Google Cloud Platform or Microsoft Azure or any other cloud provider.
1. Create Three Droplets in Digital Ocean
Sign up or log into Digital Ocean and create three Ubuntu instances with the following specifications:
- Ubunutu 16.04
- 4GB or more of RAM per instance
Note: do not select the Private networking option for your droplets. Selecting this option will prevent the setting up of the Kubernetes cluster to fail. See section Initialize the Master for more details.
2. Add a New Weave Cloud Instance
Sign up or log into Weave Cloud.
Create a new instance or rename the default instance in Weave Cloud](https://cloud.weave.works). Weave Cloud instances are the primary workspaces for your application and provides a view onto your cluster and the application that is running on it.
3. Set up a Kubernetes Cluster with kubeadm
Kubeadm is by far the simplest way to set up a Kubernetes cluster. With only a few commands, you can deploy a complete Kubernetes cluster with a resilient and secure container network onto the Cloud Provider of your choice in only a few minutes.
kubeadm
is a command line tool that and is a part of Kubernetes 1.4 and greater.
See the kubeadm
reference for information on all kubeadm
command-line options and for advice on automating kubeadm
.
Objectives
- Install a secure Kubernetes cluster
- Install Weave Net as a pod network so that application components (pods) can communicate with one another
- Install the Sock Shop, a demo microservices application
- View the result in Weave Cloud
4. Download and install kubelet, kubeadm and Docker
To begin SSH into the machine and become root
(for example, run sudo su -
). Then install the required binaries onto all three instances:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
Next, install Docker. You can also use the official Docker packages instead of docker.io
that is referenced here.
apt-get install -y docker.io
And finally, install the Kubernetes packages:
apt-get install -y kubelet kubeadm kubectl kubernetes-cni
5. Initialize the Master
Note: Before making one of your machines a master,
kubelet
andkubeadm
must have been installed onto each of the nodes.
The master is the machine where the “control plane” components run, including etcd
(the cluster database) and the API server (which the kubectl
CLI communicates with).
All of these components run in pods started by kubelet
.
Keep in mind that you can’t run kubeadm init
twice without tearing down the cluster, see Tear Down for more information.
To initialize the master, pick one of the machines on which you previously installed kubelet
and kubeadm
and run:
kubeadm init
Initialization of the master may take a few minutes.
This autodetects the network interface and then advertises the master on it with the default gateway.
Note: If you want to use a different network interface, specify it with the
--api-advertise-addresses=<ip-address>
flag when you runkubeadm init
.
Important! Special note on selecting different network interface:
When using a different network interface in Digital Ocean through the Private Network option for the droplets causes
the Kubernetes cluster set up to fail. For more information, see issue #203 of kubernetes/kubeadm. This can occur with kubeadm
and the default gateway that your droplets may receive at the moment of creation.
Until this issue is resolved, the default droplet networking settings must be enabled.
This means that all the nodes in your cluster will be open to the world and that they communicate between each other via Internet. Ensure that you understand the implications of such a set up. You can reinforce the security of your cluster by using ufw
or iptables
rules.
Refer to the kubeadm reference doc to read up on the flags kubeadm init
provides.
If the initialization is successful, the output should look similar to the following:
....some preflight checks and initialization
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token <token-id> <master-ip>
Make a record of the kubeadm join
command that kubeadm init
outputs. You will need this once it’s time to join the nodes. This token is used for mutual authentication between the master and any joining nodes.
This token is a secret, and so it’s important to keep it safe — anyone with this key can add authenticated nodes to your cluster.
(Optional) Scheduling Pods on the Master
By default, the cluster does not schedule pods on the master for security reasons. If you want to be able to schedule pods on the master, for example if you want a single-machine Kubernetes cluster for development, then run:
kubectl taint nodes --all dedicated-
The output will be:
node "test-01" tainted
taint key="dedicated" and effect="" not found.
taint key="dedicated" and effect="" not found.
This removes the “dedicated” taint from any nodes that have it, including the master node, meaning that the scheduler will then be able to schedule pods everywhere.
6. Set up the environment for Kubernetes
On the master run the following as a regular user:
sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
7. Install Weave Net as the Pod Networking Layer
In this section, you will install a Weave Net pod network so that your pods can communicate with each other.
You must add Weave Net before deploying any applications to your cluster and before kube-dns
starts up.
Note: Install only one pod network per cluster. There are two versions of the Weave Net daemonset installer. One installs Weave Net to version 1.5 of the Kubernetes binaries and the other installs to 1.6.
If you’re running Kubernetes 1.5 (and less) install Weave Net by logging onto the master and running:
kubectl apply -f https://git.io/weave-kube
If you’re running Kubernetes 1.6 (and above), install Weave Net by logging onto the master and running:
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
The output will be:
serviceaccount "weave-net" created
clusterrole "weave-net" created
clusterrolebinding "weave-net" created
daemonset "weave-net" created
Once a pod network is installed, confirm that it is working by ensuring that the kube-dns
pod is running
:
kubectl get pods --all-namespaces
Once the kube-dns
pod is up and running, you can join all of the nodes to form the cluster.
8. Join Your Nodes to the Master
The nodes are where the workloads (containers and pods, etc) run.
Join the nodes to your cluster by running:
kubeadm join --token <token> <master-ip>
The above command, including the token and the master-ip, is output by kubeadm init
that you ran earlier.
When the node has successfully joined, the output should look as follows:
preflight] Running pre-flight checks
[tokens] Validating provided token
[discovery] Created cluster info discovery client, requesting info from "http://138.197.150.135:9898/cluster-info/v1/?token-id=ad23e7"
[discovery] Cluster info object received, verifying signature using given token
[discovery] Cluster info signature and contents are valid, will use API endpoints [https://138.197.150.135:6443]
[bootstrap] Trying to connect to endpoint https://138.197.150.135:6443
[bootstrap] Detected server version: v1.6.0
[bootstrap] Successfully established connection with endpoint "https://138.197.150.135:6443"
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server:
Issuer: CN=kubernetes | Subject: CN=system:node:node-02 | CA: false
Not before: 2017-02-20 20:33:00 +0000 UTC Not After: 2018-02-20 20:33:00 +0000 UTC
[csr] Generating kubelet configuration
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
Node join complete:
* Certificate signing request sent to master and response
received.
* Kubelet informed of new secure connection details.
Run 'kubectl get nodes' on the master to see this machine join.
Run kubectl get nodes
on the master to display a cluster with the number of machines as you created.
(Optional) Control Your Cluster From Machines Other Than The Master
In order to get kubectl
on your laptop to talk to your cluster (as an example), copy the kubeconfig
file from your master to your laptop:
scp root@<master ip>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes
8. Install and Launch the Weave Cloud Agents
The yaml
file listed below installs all of the Weave Cloud probes to a DaemonSet and launches them to your cluster.
From the master:
kubectl apply -f \
"https://cloud.weave.works/k8s.yaml?t=<cloud-token>&k8s-version=$(kubectl version | base64 | tr -d '\n')"
The <cloud-token>
is found in the settings dialog on Weave Cloud. The above command may also be cut and pasted from the setup screens in Weave Cloud.
If you mistyped or copied and pasted the command incorrectly, you can remove the DaemonSet with:
kubectl delete ns weave
Return to Weave Cloud, and click Explore and then Pods to display the Kubernetes cluster in your instance. Ensure that the All Namespaces filter is enabled from the left-hand corner.
For the next few steps, keep the instance open on Explore to watch the Sock Shop containers spin up in the cluster.
9. Install the Sock Shop onto Kubernetes
To put your cluster through its paces, install the sample microservices application, Socks Shop. Learn more about the sample microservices app by referring to the microservices-demo README.
To install the Sock Shop, run the following:
kubectl create namespace sock-shop
git clone https://github.com/microservices-demo/microservices-demo
cd microservices-demo
kubectl apply -n sock-shop -f deploy/kubernetes/manifests
Click on Explore and then Pod and enable the sock-shop namespace filter from the bottom left-hand corner.
It takes several minutes to download and start all of the containers. Watch the output of kubectl get pods -n sock-shop
to see that all of the containers are successfully running.
Or view the containers as they get created in Weave Cloud.
10. View the Sock Shop in Your Browser
Find the port that the cluster allocated for the front-end service by running:
kubectl describe svc front-end -n sock-shop
The output should look like:
Name: front-end
Namespace: sock-shop
Labels: name=front-end
Selector: name=front-end
Type: NodePort
IP: 100.66.88.176
Port: <unset> 80/TCP
NodePort: <unset> 31869/TCP
Endpoints: <none>
Session Affinity: None
Launch the Sock Shop in your browser by going to the IP address of any of your node machines in your browser, and by specifying the NodePort. So for example, http://<master_ip>:<pNodePort>
. You can find the IP address of the machines in the DigitalOcean dashboard.
In the example above, the NodePort was 31869
.
If there is a firewall, make sure it exposes this port to the internet before you try to access it.
11. Create a Load on the Sock Shop
To fully appreciate the topology of the Sock Shop in Weave Scope, you’ll have to create a load on the app.
View the Sock Shop in your app with host-ip:[port number]
<host-ip:[port number]>
is the IP of the master and the port number you see when you runkubectl describe svc front-end -n sock-shop
.
With the Sock Shop displayed in the browser, log in to the application with user1
and password
. Select a few pairs of socks, put them inot the shopping cart, proceed to checkout and then return to Weave Cloud.
Click on the Containers view where you will see the app begin to take shape with lines appearing between each service.
Apply a Network Policy, and Enforce with Weave Net
In the above guide, you should have deployed the socks shop. However, the different components are not isolated.
Let’s start by testing that. Load up Weave Cloud and make sure you’re in the containers view, then observe that the catalogue service can talk to the shipping service.
Search for the “catalogue” container and then click the >_
icon from the details panel. This opens up a convenient shell for that container.
Type the following:
wget http://shipping
You should get:
wget: server returned error: HTTP/1.1 404
This is not good! The catalogue service can speak to the shipping service. For a hacker who managed to infiltrate the catalogue service, they could now get direct access to the shipping service and attack that too.
So, let’s apply some network policy. SSH into the master, or where ever you run kubectl
:
cd microservices-demo
kubectl apply -f deploy/kubernetes/manifests-policy/
Now run the wget inside the terminal in Weave Cloud again:
wget http://shipping
And you’ll see the connection just times out. Those packets are being dropped. The app is now more secure!
You can take a look at the network policy itself and learn about Kubernetes network policy to learn how to write your own policy.
Tear Down
Unless you are continuing onto another guide, or you are using the cluster for your own app, you may want to tear down the Sock Shop and the Kubernetes cluster you created.
-
To uninstall the socks shop, run
kubectl delete namespace sock-shop
on the master. -
To uninstall Kubernetes on the machines, you can delete the machines you created for this tutorial, and then start over
-
To uninstall a daemon set run
kubectl delete ds <agent-name>
.
Recreating the Cluster: Starting Over
Note: If you made an error during the install instructions, it is recommended that you delete the entire cluster and begin again.
1. Reset the cluster to the local state:
kubeadm reset
2. Run systemctl start kubelet
on each of the nodes.
3. Re-initialize the master by kubeadm init
on the master.
4. Then join the nodes to the master with:
kubeadm join --token <token> <master-ip>
Conclusions
You’ve seen that Kubernetes network policy allows you to define flexible and dynamic security policies, and Weave Net allows you to enforce them.
Join the Weave Community
If you have any questions or comments you can reach out to us on our Slack channel. To invite yourself to the Community Slack channel, visit Weave Community Slack invite or contact us through one of these other channels at Help and Support Services.