Monitoring Weave Net with Prometheus

Two endpoints are exposed: one for the Weave Net router, and, when deployed as a Kubernetes Addon, one for the network policy controller.

Router Metrics

  • weave_connections - Number of peer-to-peer connections.
  • weave_connection_terminations_total - Number of peer-to-peer connections terminated.
  • weave_ips - Number of IP addresses.
  • weave_max_ips - Size of IP address space used by allocator.
  • weave_dns_entries - Number of DNS entries.
  • weave_flows - Number of FastDP flows.
  • weave_ipam_unreachable_count - Number of unreachable peers that own IPAM addresses.
  • weave_ipam_unreachable_percentage - Percentage of all IP addresses owned by unreachable peers.
  • weave_ipam_pending_allocates - Number of pending allocates.
  • weave_ipam_pending_claims - Number of pending claims.

Kubernetes Network Policy Controller Metrics

The following metric is exposed:

  • weavenpc_blocked_connections_total - Connection attempts blocked by policy controller.

Metrics Endpoint Addresses

When installed as a Kubernetes Addon, the router listens for metrics requests on 0.0.0.0:6782 and the Network Policy Controller listens on 0.0.0.0:6781. No other requests are served on these endpoints.

Note: If your Kubernetes hosts are exposed to the public internet then these metrics endpoints will also be exposed.

When started via weave launch, by default weave listens on its local interface to serve metrics and other read-only status requests. To publish your metrics throughout your cluster, you can set WEAVE_STATUS_ADDR:

WEAVE_STATUS_ADDR=X.X.X.X:PORT

Set it to an empty string to disable.

You can also pass the parameter --metrics-addr=X.X.X.X:PORT to weave launch to specify an address to listen for metrics only.

Weave Net Monitoring Setup in Kubernetes using kube-prometheus

Weave Net monitoring can be setup in Kubernetes using the kube-prometheus library for Weave Net. You can read about the example document here.

Let’s setup weave monitoring using kube-prometheus.

Install golang

Follow this document

Install jssonet builder

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

Install jsonnet

Follow this document

Install gojsonyaml

go get github.com/brancz/gojsontoyaml

Update dependencies

jb update

Create weave-net.jsonnet

Note: Some alert configurations are environment specific and may require modifications of alert thresholds.

cat << EOF > weave-net.jsonnet
local kp =  (import 'kube-prometheus/kube-prometheus.libsonnet') +
            (import 'kube-prometheus/kube-prometheus-weave-net.libsonnet') + {
  _config+:: {
    namespace: 'monitoring',
  },
  prometheusAlerts+:: {
    groups: std.map(
      function(group)
        if group.name == 'weave-net' then
          group {
            rules: std.map(function(rule)
              if rule.alert == "WeaveNetFastDPFlowsLow" then
                rule {
                  expr: "sum(weave_flows) < 20000"
                }
              else if rule.alert == "WeaveNetIPAMUnreachable" then
                rule {
                  expr: "weave_ipam_unreachable_percentage > 25"
                }
              else
                rule
              ,
              group.rules
            )
          }
        else
          group,
        super.groups
      ),
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
EOF

Create manifests

jsonnet -J vendor -m manifests weave-net.jsonnet | xargs -I{} sh -c 'cat $1 | gojsontoyaml > $1.yaml; rm -f $1' -- {}

Apply manifests

Applying the created manifests will install the following components in your Kubernetes cluster: