At a recent Weave Online User Group (WOUG), two speakers presented topics on Kubernetes.  Sandeep Dinesh (@SandeepDinesh), Developer Advocate for Google Cloud presented a list of best practices for running applications on Kubernetes. Jordan Pellizzari (@jpellizzari), a Weaveworks engineer, followed up with a talk on lessons learned after two years of developing and running our SaaS Weave Cloud on Kubernetes.

Best Practices for Kubernetes

The best practices in this presentation grew out of discussions that Sandeep and his team had about the many different ways that you can perform the same tasks in Kubernetes. They compiled a list of those tasks and from that derived a set of best practices.

Best practices were categorized into:

  1. Building Containers
  2. Container Internals
  3. Deployments
  4. Services
  5. Application Architecture

#1 Building Containers

Don’t trust arbitrary base images!

Unfortunately, we see this happening all the time, says Pradeep. People will take a basic image from DockerHub that somebody created - because at first glance it has the package that they need - but then push the arbitrarily chosen container to production.

There’s a lot wrong with this: you could be using the wrong version of code that has exploits, has a bug in it, or worse it could have malware bundled in on purpose - you just don’t know.

To mitigate this, you can run a static analysis like CoreOS’ Clair or Banyon Collector that you can use to scan your containers for vulnerabilities.

Keep base images small

Start with the leanest most viable base image and then build your packages on top so that you know what’s inside.

Smaller base images also reduces overhead. Your app may only be about 5 mb, but if you blindly take an off-the-shelf image, with Node.js for example, it includes an extra 600MB of libraries you don’t need.

Other advantages of smaller images:

  • Faster builds
  • Less storage
  • Image pulls are faster
  • Potentially less attack surface

Use the Builder Pattern

This pattern is more useful for static languages that compile like Go and C++ or Typescript for Node.js.

In this pattern you’ll have a build container with the compiler, the dependencies and maybe unit tests. Code then runs through the first step and outputs the build artifacts. These are combined with any static files, bundles, etc. and go through a runtime container that may also contain some monitoring or debugging tools.

In the end, your Docker file should only reference your base image and the runtime env container.

#2 Container Internals

Use a non-root user inside the container

When packages are updated inside your container as root, you’ll need to change the user to a non-root user.

The reason being, if someone hacks into your container and you haven’t changed the user from root, then a simple container escape could give them access to your host where they will be root. When you change the user to non-root, the hacker needs an additional hack attempt to get root access.

As a best practice, you want as many shells around your infrastructure as possible.

In Kubernetes you can enforce this by setting the Security context runAsNonRoot: true which will make it a policy-wide setting for the entire cluster.

Make the file system read-only

This is another best practice that can be enforced by setting the option readOnlyFileSystem: true.

One process per container

You can run more than one process in a container, however it is recommend to run only one single one. This is because of the way the orchestrator works. Kubernetes manages containers based on whether a process is healthy or not. If you have 20 processes running inside a container - how will it know whether its healthy or not?

To run multiple processes that all talk and depend on one another, you’ll need to run them in Pods.

Don’t restart on failure. Crash cleanly instead

Kubernetes restarts failed containers for you, and therefore you should crash cleanly with an error code, so that they can restart successfully without your intervention.

Log everything to stdout and stderr

By default Kubernetes listens to these pipes and sends the outputs to your logging service. On Google Cloud for example they go to StackDriver logging automatically.

#3 Deployments

Use the “record” option for easier rollbacks

When applying a yaml use the --record flag:

kubectl apply -f deployment.yaml --record

With this option, everytime there is an update, it gets saved to the history of those deployments and it provides you with the ability to rollback a change.

Use plenty of descriptive labels

Since labels are arbitrary key-value pairs, they are very powerful. For example consider the diagram below with app named ‘Nifty’ spread out in four containers. With labels you can select only the backend containers by selecting the backend (BE) labels.

Use sidecars for Proxies, watchers, etc.

Sometimes you need a group of processes to communicate with one another. But you don’t want all of those to run in a single container (see above “one process per container”) and instead you would run related processes in a Pod.

Along the same lines is when you are running a proxy or a watcher that your processes depend on. For example, a database that your processes depend on. You wouldn’t hardcode the credentials into each container. Instead you can deploy the credentials as a proxy into a sidecar where it securely handles the connection:

Don’t use sidecars for bootstrapping!

Although sidecars are great for handling requests both outside and inside the cluster, Sandeep doesn’t recommend using them for bootstrapping. In the past bootstrapping was the only option, but now Kubernetes has ‘init containers’.

In the case of a process running in one container that is dependant on a different microservice, you can use `init containers` to wait until both processes are running before starting your container. This prevents a lot of errors from occurring when processes and microservices are out of sync.

Basically the rule is: use sidecars for events that always occur and use init containers for one time occurrences.

Don’t use :latest or no tag

This one is pretty obvious and most people doing this today already. If you don’t add a tag for your container, it will always try to pull the latest one from the repository and that may or may not contain the changes you think it has.

Readiness and Liveness Probes are your friend

Probes can be used so that Kubernetes knows if nodes are healthy and if it should send traffic to it. By default Kubernetes checks if processes are running or not running. But by using probes you can leverage this default behaviour in Kubernetes to add your own logic.

#4 Services

Don’t use type: LoadBalancer

Whenever you add load balancer to your deployment file on one of the public cloud providers, it spins one up. This is great for High Availability and speed, but it costs money.

Tip: Use Ingress instead which lets you load balance multiple services through a single end-point. This is not only simpler, but also cheaper.

This strategy of course will only work if you doing http or web stuff and it won’t work for UDP or TCP based applications.

Type: Nodeport can be “good enough”

This is more of a personal preference and not everyone recommends this. NodePort exposes your app to the outside world on a VM on a particular port. The problem with it is it may not be as highly available as a load balancer. For example, if the VM goes down so does your service.

Use Static IPs they are free!

On Google Cloud this is easy to do by creating Global IPs for your ingress’. Similarly for your load balancers you can use Regional IPs. In this way, when your service goes down you don’t have to worry about your IPs changing.

Map External Services to Internal Ones

This is something that most people don’t know you can do in Kubernetes. If you need a service that is external to the cluster, what you can do is use configMaps to map the name of a service to your cluster or to a Pod. Now you can just call the service by its name and the Kubernetes manager passes you on to it as if it’s part of the cluster. Kubernetes treats the service as is if it is on the same network, but it sits actually outside of it.

#5 Application Architecture

Use Helm Charts

Helm is basically a repository for packaged up Kubernetes configurations. If you want to deploy a MongoDB. There’s a preconfigured Helm chart for it with all of its dependencies that you can easily use to deploy it to your cluster.

There are many Helm charts for popular software components that will save you a lot of time and effort.

All Downstream dependencies are unreliable

Your application should have logic and error messages in it to account for any dependencies over which you have no control. To help you with the downstream management, Sandeep suggests, you can use a service mesh like Istio or Linkerd.

Use Weave Cloud

Clusters are difficult to visualize and manage and so using Weave Cloud really helps you see what’s going on inside and to keep track of dependencies.

Make sure your microservices aren’t too micro

You want logical components and not every single function turned into a microservice.

Use Namespaces to split up your cluster

For example you can create Prod, Dev and Test in the same cluster with different namespaces and also use namespaces to limit the amount of resources so that one buggy process doesn’t use all of the cluster resources.

Role based Access Control

Enact proper access control to limit the amount of access to your cluster as a best practices security measure.

Lessons Learned from Running Weave Cloud in Production

Next Jordan Pellizzari spoke what we’ve learned from running and developing Weave Cloud on Kubernetes for the past two years.

We currently run on AWS EC2 and have about 72 Kubernetes Deployments running on 13 hosts and across about 150 containers.

All of our persistent storage is kept in S3, DynamoDB or RDS, and we don’t keep state in containers.

For more details on how we set up our infrastructure refer to Weaveworks & AWS: How we manage Kubernetes clusters.

Challenge 1. Version Control for Infrastructure

At Weaveworks all of our infrastructure is kept in Git, and when we make an infrastructure change, like code, it is also done via pull request. We’ve been calling this GitOps and we have a number of blogs about it. You can start with the first one: GitOps - Operations by Pull Request”.

At Weave, Terraform scripts, Ansible and of course Kubernetes YAML files are all in Git under version control.

There’s a number of reasons that it’s best practice to keep your infrastructure in Git:

  • Releases are easily rolled back
  • An auditable trail of who did what is created
  • Disaster recovery is much simpler

Problem: What do you do when Prod doesn’t match Version Control?

In addition to keeping everything in Git, we also run a process that checks the differences between what’s running in the prod cluster with what’s in checked into Version Control. When it detects a difference, an alert is sent to our slack channel.

We check differences with our open source tool called Kube-Diff.

Challenge 2. Automating Continuous Delivery

Automate your CI/CD pipeline and avoid manual Kubernetes deployments. Since we are deploying multiple times a day, this approach saves the team valuable time as it removes manual error prone steps. At Weaveworks, developers simply do a Git push and Weave Cloud takes care of the rest:

  • Tagged Code runs through CircleCI tests and builds a new container image and pushes the new image to the registry.
  • The Weave Cloud ‘Deploy Automator’ notices the image, pulls the new image from the repository and then updates its YAML in the config repo.  
  • The Deploy Synchronizer, detects that the cluster is out of date, and it pulls the changed manifests from the config repo and deploys the new image to the cluster.

Here is a longer article (The GitOps Pipeline) on what we believe to be the best practices when building an automated CICD pipeline.

In Summary

Sandeep Dinesh provided an in-depth overview of 5 Best Practices for creating, deploying and running applications on Kubernetes. This was followed by a talk by Jordan Pellizzari on how Weave manages its SaaS product Weave Cloud in Kubernetes and the lessons learned.

Watch the video in its entirety:

You may also be interested in reading our blog, "Kubernetes Beginner's Guide: Learning the basics in an hour"

For more talks like these, join the Weave Online User Group.