10 Tips for Building and Managing Containers
Learn how your team can speed up development with these hands-on tips for building and managing containerized apps on Kubernetes.
Learn how your team can speed up development with these hands-on tips for building and managing containerized apps on Kubernetes.
With Kubernetes, you can scale your business automatically and immediately with little to no downtime. As a result, you can optimize IT costs and increase the reliability of your systems.
Containers are at the center of an application running in Kubernetes. When you create Kubernetes workloads (the rules for scheduling, scaling, and upgrading an application), you start with a container image that runs your service or Kubernetes workload. After the image is tested and integrated with the rest of the application’s code base, it is generally pushed to a container registry. Before we get to that point, there are many best practices to keep in mind when you are writing your services and containerizing them. Here are our best tips and tricks for managing containers:
#1 Keep up with the latest Kubernetes patterns
With new features being released continuously, Kubernetes usage patterns can change. To make sure that your cluster follows the current Kubernetes usage pattern, we encourage you to follow and periodically read the official Kubernetes documentation as well as the changes introduced into each Kubernetes release in the release notes.
#2 Save time and curate base images
Creating application containers to use with your Kubernetes cluster involves building a Docker base image that some or all application containers are based upon. Using a base image makes it possible to reuse image configurations since many apps will share dependencies, libraries, and configurations.
Docker Hub or Google Container Registry have thousands of ready-to-use base images available for download. You can save a lot of time by using these pre-configured base images for your apps.
#3 Don’t trust arbitrary base images. Always use a vulnerability scanner.
Although it’s convenient, there are risks associated with using pre-built images. We recommend running some sort of vulnerability scan to be safe.
At first glance, it may look like a basic image created by someone else has the package you need. Some developers will take that basic image from DockerHub that somebody else created and then push the arbitrarily chosen container to production.
Here’s why this is wrong: you could be using the wrong version of code that has exploits, it could have a bug in it, or worse it could have malware bundled in on purpose.
To mitigate this, you can run a static analysis like Snyk or Twistlock and incorporate it into your CICD pipeline. This ensures that all of your containers are being scanned for vulnerabilities.
As a general rule, if you do find a vulnerability in your base image, you should rebuild it in addition to patching it. Containers are meant to be immutable. As a result, the best practice is to rebuild the image with the included patches and redeploy it.
#4 Optimize base images
Start with the leanest and most viable base image and then build your packages on top of that. This way, you will know exactly what is inside your container.
Smaller base images also reduce any overhead. Your app may only be 5MB, but if you add an off-the-shelf image like Node.js and include the entire library, you could end up with 600MB of extra libraries that you don’t need.
Other advantages of smaller images include the following:
- Faster builds
- Less storage
- Faster image pulls
- Potential less attack surface
#5 Use only one process per container
In addition to keeping base images small, only have one process per container. A container has the same lifecycle as the app it hosts, which means that each of your containers should only contain a single parent process.
According to Google Cloud, it’s a common mistake to treat containers like virtual machines which can run multiple processes simultaneously. While it is possible to use containers in this way, it doesn’t take advantage of Kubernetes self-healing properties.
As a rule, both the container and the app should start and stop at the same time. If you have multiple processes in one container, you might experience mix-ups within the application which can result in Kubernetes being unable to determine if a container is healthy or not.
#6 Properly handle Linux signals
Linux signals are used to control the lifecycle of processes inside of a container. In order to link the lifecycle of your app to the container, you need to ensure that your app properly handles Linux signals.
Linux signals such as SIGTERM, SIGKILL, and SIGINT are used within the kernel to terminate processes. Inside the container, Linux executes these common signals differently. They don’t function as expected, resulting in errors and interrupted writes.
A way to overcome this is to use a specialized init system that is suited to containers, like Linux Tini. This tool correctly registers signal handlers (such as PIDs) so that Linux signals will work properly for containerized applications by shutting down orphaned and zombie processes in order to recoup the memory.
#7 Take advantage of the Docker to build cache order
Container images are built in a series of layers using instructions found in a template or a Dockerfile. The layers and the order in which they are built are typically cached by the container platform. For example, Docker has a build cache that allows layer reuse. This cache makes your builds much faster, but you’ll only be able to use it if all layers used in a previous build are present.
For example, let’s say you have a build file with step X, step Y, and step Z and you make a change to step Z. In this case, the build file reuses steps X and Y from the cache since those layers were present before the layer you changed. This saves you time by accelerating the build. If you only changed X, the cache will not include the layers nor any steps after it.
While this is convenient behavior, you have to ensure that all of your layers are up to date and are not being drawn from an out of date cache.
#8 Use a package manager like Helm
As an unofficial package manager for Kubernetes, Helm is another option for sourcing and updating common workloads and containers to run on your cluster. For custom applications, Helm uses a chart to declare dependencies and provides tools for rolling upgrades and rollbacks.
You can take advantage of existing base images for generic services that you would want to provide inside your Kubernetes cluster, like databases or web servers. An alternative option is to create custom base images for your in-house applications. Creating your own charts will make deployments simpler with less overhead and duplication of work for your development team.
For a tutorial on how Helm works, see, "Managing Helm Releases the GitOps Way"
#9 Use tags and semantic versioning
As a general rule, you shouldn’t use the latest tag. This is pretty obvious for most developers, but if you don’t add a custom tag for your container, it will always try to pull the latest one from the repository. Unfortunately, you cannot be certain that the latest container holds the changes you expect.
When creating custom images, use image tagging and semantic versioning to keep track of changes to your Docker containers. When they run in Kubernetes, image tags are how you communicate which version of the image you want to run on your Kubernetes cluster. For optimal results with Kubernetes, you should pick a Docker image versioning scheme with both the production workload and the development flow in mind.
#10 Manage secrets wisely
In many cases when building Docker images, you need to give the applications running inside the container access to sensitive data. Those could include API tokens, private keys, and database connection strings.
Embedding those secrets inside the containers is not a secure solution, even if you are keeping the images private. Pushing an unencrypted secret as part of a Docker image exposes you to a myriad of additional security risks, including the security of your network and your image registry. The Docker architecture itself is not optimized for allowing unencrypted sensitive data inside the containers.
Instead, the easiest way to securely store the secrets outside your containers is to use Kubernetes Secrets objects.
Kubernetes offers a Secrets abstraction which allows you to store the secrets outside of the Docker images or the pod definitions. You can expose Kubernetes secrets as mounted volumes inside the containers or as environment variables. When you update the secret values in Kubernetes Secrets, it's enough to rotate the pods of your service to start using the new credentials. There are other options available for storing secrets, such as Hashicorp Vault and Bitnami sealed secrets.
Interested in learning more? Read more about managing containers here.
