To begin reporting metrics, you must install the Weave Cloud agents to your Kubernetes cluster. The installed Prometheus agent will, by default:

  • Discover and scrape all pods running in the cluster.
  • Scrape system components: API server, kubelet and cAdvisor.
  • Export information about Kubernetes objects with kube-state-metrics.
  • Add two synthetic labels _weave_namespace and _weave_service to every metric.
  • Add another synthetic label, _weave_pod_name, to cAdvisor metrics.
  • Push the discovered metrics to Weave Cloud.

Synthetic Labels

Our default configuration adds some synthetic labels to help with querying data back from Weave Cloud and building nice dashboards.

Every metric comes with two additional labels computed at scrape time:

  • _weave_namespace is the Kubernetes namespace of the pod the metric comes from. This label can be used to distinguish between the same component (eg. consul) running in two separate namespaces.
  • _weave_service is a human friendly name derived from the pod name the metric comes from. For instance, if a pod is created from a Deployment with the name billing-1935513387-8jtcc, _weave_service will be set to billing. In this context, service isn’t a Kubernetes service but the name derived from a pod which, in-turn, is derived from the name of the controller that created the pod.

In addition, a _weave_pod_name label is added to the container_* metrics scraped from cAdvisor. This label is generated from the pod_name label using the same rules as _weave_service above. _weave_pod_name can be used in legends of dashboard graphs with container level metrics. As an example of cAdvisor query using _weave_pod_name, we can visualize CPU usage grouped by service with:

sum by (_weave_pod_name) (rate(container_cpu_usage_seconds_total{image!=""}[5m])

Per-pod Prometheus Annotations

Annotations on pods allow a fine control of the scraping process:

  • prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
  • prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
  • prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared).

These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets. The DaemonSet manifest below will instruct Prometheus to scrape all of its pods on port 9102.

apiVersion: apps/v1beta2 # for versions before 1.8.0 use extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9102'
    spec:
      containers:
      - name: fluentd-elasticsearch
        image: gcr.io/google-containers/fluentd-elasticsearch:1.20

Default Scraping Policy

To minimize the amount of configuration required from the user, the default setup will try to scrape all pods present in the system. This may have some undesirable side-effects such as:

  • Prometheus issuing HTTP GET requests to TCP ports not even exposing an HTTP server.
  • Prometheus issuing HTTP GET requests to fetch /metrics on services that may not export any metric, polluting logs.

While the annotations described in the previous section allow the user to opt-out of the scraping process, we also offer an alternative Prometheus configuration that only scrapes a minimal set of system metrics and requires the user to opt-in on a per-pod basis. In other words, in this mode the prometheus.io/scrape: true annotation is required for Prometheus to scrape the pod.

To apply the opt-in configuration to your cluster, use the cortex-scrape-policy=opt-in parameter:

kubectl apply -f \
    "https://cloud.weave.works/k8s.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')&cortex-scrape-policy=opt-in&t=<cloud-token>"

Where,

  • <cloud-token> is the token you obtained when you signed up for Weave Cloud.

Further Reading