PromQL queries for the rest of us
Prometheus comes with its own query language called PromQL. It’s very powerful and easily allows you to filter with the multi-dimensional time-series labels that make Prometheus so great. But it can be daunting when you are faced with an...
Prometheus comes with its own query language called PromQL. It’s very powerful and easily allows you to filter with the multi-dimensional time-series labels that make Prometheus so great. But it can be daunting when you are faced with an empty input field to form a query while your system is on fire. Enter the UI engineers to help you out.
Anatomy of a query language
If you look at monitoring for our own system, Weave Cloud, you come across queries like the following:
sum(rate(container_cpu_user_seconds_total{
namespace="cortex",
image=~"quay.io/weaveworks/cortex.+"
}[1m])) by (instance)
Looks scary, doesn’t it–especially if you are supposed to come up with queries like that on your own. The core part of any PromQL query are the metric names of a time-series, e.g., container_cpu_user_seconds_total and container_cpu_system_seconds_total. The suffix _seconds_total tells us that the metric is an accumulator with its unit being seconds. A second-accumulator can get us to per-second usage if we take its rate(), since that returns the metric’s per-second values. If we now ran rate(container_cpu_user_seconds_total[1m]), this would return us ~1200 rows of CPU usage for each running entity that emits that metric, no matter where they run. We need some help to narrow down our search and aggregate the results.
A helping hand
Let’s say we want to focus on Cortex’s resource usage itself, especially the CPU consumption per pod as it runs on a Kubernetes cluster. First, let’s make sure we only look at time-series that run on a pod. Cortex’s query field is smart enough to figure out the label dimensions from a metric name like container_cpu_user_seconds_total. As we type the opening curly braces for PromQL’s filter part, the query field requests the label space and offers us a small list of label keys. pod_name looks pretty good, as things do not run in a Kubernetes pod will not have a pod name. We have our first filter!
rate(container_cpu_user_seconds_total{pod_name!=""}[1m])
We are now down to ~600 rows. Not only have we halved our search space, we also learned a bit more about the data: we can see which labels are further available for pods, as their cells in the table are not empty.
The next question is which of those entities are relevant for Cortex resource usage. Again we turn to the filter suggestions and quickly type namespace and choose cortex from the label value suggestions. (Since the roll-out of the new values tables, you can also click on a cortex cell in the table to apply this filter.)
rate(container_cpu_user_seconds_total{
namespace="cortex",
pod_name!=""
}[1m])
Down to ~100 rows. Gee, Kubernetes sure is a beast. The interesting bits are still lying ahead: Cortex runs a couple of supporting processes, e.g., Consul, but we would like to know only about the services we wrote. Here the image label can help, as it encodes which container image a pod is running. At Weaveworks, our Cortex-related image names usually start with quay.io/weaveworks/cortex..., this shall make a great next filter:
rate(container_cpu_user_seconds_total{
namespace="cortex",
image=~"quay.io/weaveworks/cortex.+"
}[1m])
At this point we’re down to ~25 rows, and by the looks of it, we are only including relevant rows.
Give me the usage already!
We can now use the power of computers of sum up those rates for us. But wait, let’s look at the table again: Notice the instance column? We are dealing with multiple machines here, so we want to include those in the breakdown:
sum(rate(container_cpu_user_seconds_total{
namespace="cortex",
image=~"quay.io/weaveworks/cortex.+"
}[1m])) by (instance)
Here we go: four rows! One for each machine we are running. Time to graph it:
That was not so difficult, was it?
Bonus: Blue/green usage
The Cortex developers have been busy on Friday, time to upgrade our ingesters. Monday morning the roll-out went well, but now something is odd and we are getting paged about CPU usage. The working theory is that the new ingesters are using much more CPU. Really? PromQL to the rescue! Let’s apply our new skills, filter like for ingesters and, you have guessed it, have a different breakdown, this time by image;
sum(rate(container_cpu_user_seconds_total{
namespace="cortex",
image=~"quay.io/weaveworks/cortex-ingester.+"
}[1m])) by (instance, image)
Phew, CPU usage is the same for old and new images. We are off the hook. Take that, incident responder!
