To begin reporting metrics, you must install the Weave Cloud agents to your Kubernetes cluster. The installed Prometheus agent will, by default:
- Discover and scrape all pods running in the cluster.
- Scrape system components: API server, kubelet and cAdvisor.
- Export information about Kubernetes objects with kube-state-metrics.
- Add three synthetic labels
kubernetes_namespace
,kubernetes_pod_name
and_weave_service
to every metric. - Add another synthetic label,
_weave_pod_name
, to cAdvisor metrics. - Push the discovered metrics to Weave Cloud.
Synthetic Labels
Our default configuration adds some synthetic labels to help with querying data back from Weave Cloud and building nice dashboards.
Every metric comes with two additional labels computed at scrape time:
kubernetes_namespace
is the Kubernetes namespace of the pod the metric comes from. This label can be used to distinguish between the same component (eg. consul) running in two separate namespaces.kubernetes_pod_name
is the name of the pod the metric comes from. This label can be used to distinguish between metrics from different pods of the same Deployment or DaemonSet._weave_service
is a human friendly name derived from the pod name the metric comes from. For instance, if a pod is created from a Deployment with the namebilling-1935513387-8jtcc
,_weave_service
will be set tobilling
. In this context, service isn’t a Kubernetes service but the name derived from a pod which, in-turn, is derived from the name of the controller that created the pod.
In addition, a _weave_pod_name
label is added to the container_*
metrics scraped from cAdvisor. This label is generated from the pod_name
label using the same rules as _weave_service
above. _weave_pod_name
can be used in legends of dashboard graphs with container level metrics. As an example of cAdvisor query using _weave_pod_name
, we can visualize CPU usage grouped by service with:
sum by (_weave_pod_name) (rate(container_cpu_usage_seconds_total{image!=""}[5m])
Per-pod Prometheus Annotations
Annotations on pods allow a fine control of the scraping process:
prometheus.io/scrape
: The default configuration will scrape all pods and, if set tofalse
, this annotation will exclude the pod from the scraping process.prometheus.io/path
: If the metrics path is not/metrics
, define it with this annotation.prometheus.io/port
: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared).
These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets. The DaemonSet manifest below will instruct Prometheus to scrape all of its pods on port 9102.
apiVersion: apps/v1beta2 # for versions before 1.8.0 use extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: weave
labels:
app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9102'
spec:
containers:
- name: fluentd-elasticsearch
image: gcr.io/google-containers/fluentd-elasticsearch:1.20
Further Reading