Top 5 Kubernetes Best Practices
Learn 5 best practices for building, deploying and managing services in Kubernetes from Sandeep Dinesh, Developer Advocate, Google Cloud Platform.
From Sandeep Dinesh (Google)
Learn 5 best practices for building, deploying and managing services in Kubernetes from Sandeep Dinesh, Developer Advocate, Google Cloud Platform.
At a recent Weave Online User Group (WOUG), two speakers presented topics on Kubernetes. Sandeep Dinesh (@SandeepDinesh), Developer Advocate for Google Cloud presented a list of best practices for running applications on Kubernetes. Jordan Pellizzari (@jpellizzari), a Weaveworks engineer, followed up with a talk on lessons learned after two years of developing and running our SaaS Weave Cloud on Kubernetes.
What is Kubernetes?
Check out this primer guide where we provide an overview of what Kubernetes is and how it allows fast growing applications to scale quickly.
Kubernetes Best Practices
The best practices in this presentation grew out of discussions that Sandeep and his team had about the many different ways that you can perform the same tasks in Kubernetes. They compiled a list of those tasks and from that derived a set of best practices.
Kubernetes Best practices were categorized into:
- Building Containers
- Container Internals
- Deployments
- Services
- Application Architecture
#1 Building Containers
Don’t trust arbitrary base images!
Unfortunately, we see this happening all the time, says Pradeep. People will take a basic image from DockerHub that somebody created — because at first glance it has the package that they need — but then push the arbitrarily-chosen container to production.
There’s a lot wrong with this: you could be using the wrong version of code that has exploits, has a bug in it, or worse it could have malware bundled in on purpose - you just don’t know.
To mitigate this, you can run a static analysis like CoreOS’ Clair or Banyon Collector that you can use to scan your containers for vulnerabilities.
Keep base images small
Start with the leanest most viable base image and then build your packages on top so that you know what’s inside.
Smaller base images also reduce overhead. Your app may only be about 5 mb, but if you blindly take an off-the-shelf image, with Node.js for example, it includes an extra 600MB of libraries you don’t need.
Other advantages of smaller images:
- Faster builds
- Less storage
- Image pulls are faster
- Potentially less attack surface
Use the Builder Pattern
This pattern is more useful for static languages that compile like Go and C++ or Typescript for Node.js.
With this pattern, you’ll have a build container with the compiler, the dependencies, and maybe unit tests. Code then runs through the first step and outputs the build artifacts. These are combined with any static files, bundles, etc. and go through a runtime container that may also contain some monitoring or debugging tools.
In the end, your Docker file should only reference your base image and the runtime env container.
#2 Container Internals
Use a non-root user inside the container
When packages are updated inside your container as root, you’ll need to change the user to a non-root user.
The reason is, if someone hacks into your container and you haven’t changed the user from the root, then a simple container escape could give them access to your host where they will be root. When you change the user to non-root, the hacker needs an additional hack attempt to get root access.
As a best practice, you want as many shells around your infrastructure as possible.
In a Kubernetes environment you can enforce this by setting the Security context runAsNonRoot: true which will make it a policy-wide setting for the entire cluster.
Make the file system read-only
This is another best practice that can be enforced by setting the option readOnlyFileSystem: true.
One process per container
You can run more than one process in a container, however, it is recommended to run only a single process. This is because of the way the orchestrator works. Kubernetes manages containers based on whether a process is healthy or not. If you have 20 processes running inside a container - how will it know whether it's healthy or not?
To run multiple processes that all talk and depend on one another, you’ll need to run them in Pods.
Don’t restart on failure. Crash cleanly instead
Kubernetes restarts failed containers for you, and therefore you should crash cleanly with an error code so your containers can restart successfully without your intervention.
Log everything to stdout and stderr
By default Kubernetes listens to these pipes and sends the outputs to your logging service. On Google Cloud for example they go to StackDriver logging automatically.
#3 Deployments
Use the “record” option for easier rollbacks
When applying a yaml use the --record flag:
kubectl apply -f deployment.yaml --record
With this option, every time there is an update, it gets saved to the history of those deployments and it provides you with the ability to roll back a change.
Use plenty of descriptive labels
Since labels are arbitrary key-value pairs, they are very powerful. For example, consider the diagram below with an app named ‘Nifty’ spread out in four containers. With labels, you can select only the backend containers by selecting the backend (BE) labels.
Use sidecars for Proxies, watchers, etc.
Sometimes you need a group of processes to communicate with one another. But you don’t want all of those to run in a single container (see above “one process per container”) and instead you would run related processes in a Pod.
Along the same lines is when you are running a proxy or a watcher that your processes depend on. For example, a database that your processes depend on. You wouldn’t hardcode the credentials into each container. Instead, you can deploy the credentials as a proxy into a sidecar where it securely handles the connection:
Don’t use sidecars for bootstrapping!
Although sidecars are great for handling requests both outside and inside the cluster, Sandeep doesn’t recommend using them for bootstrapping. In the past bootstrapping was the only option, but now Kubernetes has ‘init containers’.
In the case of a process running in one container that is dependent on a different microservice, you can use `init containers` to wait until both processes are running before starting your container. This prevents a lot of errors from occurring when processes and microservices are out of sync.
Basically the rule is: use sidecars for events that always occur and use init containers for one time occurrences.
Don’t use :latest or no tag
This one is pretty obvious and most people are doing this today already. If you don’t add a tag for your container, it will always try to pull the latest one from the repository and that may or may not contain the changes you think it has.
Readiness and Liveness Probes are your friend
Probes can be used so that Kubernetes knows if nodes are healthy and if it should send traffic to it. By default Kubernetes checks if processes are running or not running. But by using probes you can leverage this default behavior in Kubernetes to add your own logic.
Did you know that you can automate and streamline your software deployments with GitOps? Read this blog to learn more “ GitOps - The Path to A Fully-Automated CI/CD Pipelines.”
#4 Services
Don’t use type: LoadBalancer
Whenever you add a load balancer to your deployment file on one of the public cloud providers, it spins one up. This is great for High Availability and speed, but it costs money.
Tip: Use Ingress instead which lets you load balance multiple services through a single end-point. This is not only simpler but also cheaper.
This strategy, of course, will only work if you are doing http or web stuff and it won’t work for UDP or TCP-based applications.
Type: Nodeport can be “good enough”
This is more of a personal preference and not everyone recommends this. NodePort exposes your app to the outside world on a VM on a particular port. The problem with it is it may not be as highly available as a load balancer. For example, if the VM goes down so does your service.
Use Static IPs; they are free!
On Google Cloud this is easy to do by creating Global IPs for your ingress’. Similarly for your load balancers, you can use Regional IPs. In this way, when your service goes down you don’t have to worry about your IPs changing.
Map External Services to Internal Ones
This is something that most people don’t know you can do in Kubernetes. If you need a service that is external to the cluster, what you can do is use a service with the type ExternalName. Now you can just call the service by its name and the Kubernetes manager passes you on to it as if it’s part of the cluster. Kubernetes treats the service as is if it is on the same network, but it actually sits outside of it.
#5 Application Architecture
Use Helm Charts
Helm is basically a repository for packaged-up Kubernetes configurations. If you want to deploy a MongoDB. There’s a preconfigured Helm chart for it with all of its dependencies that you can easily use to deploy it to your cluster.
There are many Helm charts for popular software components that will save you a lot of time and effort.
All Downstream dependencies are unreliable
Your application should have logic and error messages in it to account for any dependencies over which you have no control. To help you with the downstream management, Sandeep suggests, you can use a service mesh like Istio or Linkerd.
Make sure your micro-services aren’t too micro
You want logical components and not every single function turned into a micro-service.
Kubernetes namespace best practices
Use Namespaces to split up your cluster. For example, you can create Prod, Dev and Test in the same cluster with different namespaces. Namespaces can also be used to limit the number of resources so that one buggy process doesn’t use all of the cluster’s resources.
Role based Access Control
Enact proper access control to limit the amount of access to your cluster as a best practices security measure using role-based access control (RBAC).
Lessons Learned from Running Weave Cloud in Production
Next, Jordan Pellizzari spoke about what we’ve learned from running and developing Weave Cloud on Kubernetes for the past two years.
We currently run on AWS EC2 and have about 72 Kubernetes Deployments running on 13 hosts and across about 150 containers.
All of our persistent storage is kept in S3, DynamoDB or RDS, and we don’t keep state in containers.
For more details on how we set up our infrastructure refer to Weaveworks & AWS: How we manage Kubernetes clusters.
Challenge 1. Version Control for Infrastructure
At Weaveworks all of our infrastructure is kept in Git, and when we make an infrastructure change, like code, it is also done via pull request. We’ve been calling this GitOps and we have a number of blogs about it. You can start with the first one: GitOps - Operations by Pull Request”.
At Weaveworks, Terraform scripts, Ansible and of course Kubernetes YAML files are all in Git under version control.
There’s a number of reasons that it’s best practice to keep your infrastructure in Git:
- Releases are easily rolled back
- An auditable trail of who did what is created
- Disaster recovery is much simpler
Problem: What do you do when Prod doesn’t match Version Control?
In addition to keeping everything in Git, we also run a process that checks the differences between what’s running in the prod cluster with what’s checked into Version Control. When it detects a difference, an alert is sent to our slack channel.
We check differences with our open-source tool called Kube-Diff.
Challenge 2. Automating Continuous Delivery
Automate your CI/CD pipeline and avoid manual Kubernetes deployments. Since we are deploying multiple times a day, this approach saves the team valuable time as it removes manual error-prone steps. At Weaveworks, developers simply do a Git push and Weave Cloud takes care of the rest:
- Tagged Code runs through CircleCI tests and builds a new container image and pushes the new image to the registry.
- The Weave Cloud ‘Deploy Automator’ notices the image, pulls the new image from the repository and then updates its YAML in the config repo.
- The Deploy Synchronizer, detects that the cluster is out of date, and it pulls the changed manifests from the config repo and deploys the new image to the cluster.
Here is a longer article (The GitOps Pipeline) on what we believe to be the best practices when building an automated CICD pipeline.
Don't Forget Overlooked Best Practices
After considering all insights provided by Sandeep Dinesh and Jordan Pellizzari, an easily overlooked task is making sure your Kubernetes version is up to date. You must always update your Kubernetes to have the latest stable Kubernetes version in your production cluster. The latest version will provide new features, include vulnerability and security fixes, and overall enables you to avoid security, performance or costs that could harm your service. Older versions don’t get enough support from vendors, providers or the community. It’s a best practice to update the cluster on the latest version of Kubernetes.<h2>In Summary</h2>
Sandeep Dinesh provided an in-depth overview of 5 Best Practices for creating, deploying and running applications on Kubernetes. This was followed by a talk by Jordan Pellizzari on how Weave manages its SaaS product Weave Cloud in Kubernetes and the lessons learned.
Watch the video in its entirety:
You may also be interested in reading our blog, "Kubernetes Beginner's Guide: Learning the basics in an hour"
For more talks like these, join the Weave Online User Group.
Whitepaper: Automating Kubernetes with GitOps
In this whitepaper we look at operating models for building cloud-native applications, the key benefits of GitOps, handling configuration updates with Kubernetes, and more.
