Log Aggregation for Kubernetes with Loggly
I consider log aggregation to be one of “pillars” of observability – a crucial service that every developer needs when working with distributed apps / microservices / cloud natives. In this post I’ll explain how we uses Loggly to aggregate...
I consider log aggregation to be one of “pillars” of observability – a crucial service that every developer needs when working with distributed apps / microservices / cloud natives. In this post I’ll explain how we uses Loggly to aggregate the logs from the services behind Weave Cloud. I’ll explain how our Kubernetes Pod logs get into Loggly, and some basic how to usage of Loggly.
Getting Pod Logs into Loggly
Getting Pod logs into Loggly was surprising easy, although getting the Kubernetes metadata required a bit of sleuthing. Loggly provide a plugin for fluentd to send logs to Loggly – enabling this is as easy as adding the following to your fluentd.conf (which is, of course, managed as a Kubernetes ConfigMap):
<match **> @type loggly loggly_url https://logs-01.loggly.com/inputs/<REDACTED>/tag/fluentd </match>
And ensuring the fluentd container has the appropriate gem installed (our fluent-docker container is built from
a Dockerfile in our private monitoring repo):
gem install fluent-plugin-loggly
You’ll also need to “teach” fluentd where Kubernetes keeps the container logs. TLDR; the Kubelet on each node symlinks the Docker logs into /var/log/containers/, and puts the Pod UUID in the filename. More on this later.
As implied by the mention of the Kubelet, Kubernetes keeps logs for each container locally to the node that container is running on – nothing is shipping them around the cluster for you. As such, fluentd needs to be run on each node, as a DaemonSet:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd-loggly
spec:
template:
metadata:
labels:
name: fluentd-loggly
spec:
containers:
- name: fluentd-loggly
image: quay.io/weaveworks/fluentd-loggly
volumeMounts:
- name: config-volume
mountPath: /fluentd/etc
- name: varlog
mountPath: /var/log
- name: mntcontainersdockercontainers
mountPath: /mnt/containers/docker/containers
readOnly: true
volumes:
- name: config-volume
configMap:
name: fluentd-loggly
- name: varlog
hostPath:
path: /var/log
- name: mntcontainersdockercontainers
hostPath:
path: /mnt/containers/docker/containers
You’ll notice this Pod mounts two directories from the host filesystem: /var/log and /mnt/containers/docker/containers. This is because we have our Docker logs configured to be written to our EC2 ephemeral storage, which is mounted at /mnt/containers. The symlinks managed by the Kubelet in /var/log/containers point to /mtn/containers/..., and that path need to exist inside the container – Docker/Kubernetes doesn’t magically make this happen for you.
With this setup, logs will be flowing into Loggly – but we want to be able to slice and dice our logs by the Pod’s name, service, namespace and labels. I mentioned the Kubelet puts the Pod UUID in the filename – this is to enable a plugin to lookup the Pod in the Kubernetes API and enrich the log with Kubernetes metadata. To set this up, you need to add the Kubernetes metadata filter to the fluentd config:
<filter kubernetes.var.log.containers.**.log> @type kubernetes_metadata </filter>
And ensure the gem is installed:
gem install fluent-plugin-kubernetes_metadata_filter
Using Loggly
In the Loggly UI, the tab you’ll spend most of your time in is “Search”:
Logs are sent as JSON objects. The treeview component on the left allows you to navigate the JSON objects, and pick values to filter fields by – for instance, to see all the logs for a given namespace, pick kubernetes > namespace_name and select the namespace you want:
Notice a filter is put into the “Applied Filters:” bar – you’ll need to remove these periodically, as they tend to pile up.
The central “view” shows the log events, and a bar graph of number of log events over time:
This isn’t a particularly useful view – who wants to see all the JSON events being posted? You’ll want to select the grid view button (second on the left above the JSON documents) and set this up just to show the time and log line, using the “+ Add column to grid” button:
Other than this, the search box is quite useful – for instance, you can easily see which containers have panic’d, when and how often:
Thank you for reading our blog. We build Weave Cloud, which is a hosted add-on to your clusters. It helps you iterate faster on microservices with continuous delivery, visualization & debugging, and Prometheus monitoring to improve observability.
Try it out, join our online user group for free talks & trainings, and come and hang out with us on Slack.
