Observability beyond logging for Java Microservices

Monitoring distributed applications is best approached using a combination of tools. Luke Marsden describes how Prometheus, openTracing and Weave Cloud visualization cover the bases to establish root cause of problems in distributed applications.

Weave Cloud UI Alerts and Metrics

After a particularly egregious outage was reported by a Weave Cloud user, we decided it would be a good idea to add some alerting to our front-end codebase. Alerting and metrics for Weave Cloud are handled using Prometheus, however, which...

PromQL queries for the rest of us

Prometheus comes with its own query language called PromQL. It’s very powerful and easily allows you to filter with the multi-dimensional time-series labels that make Prometheus so great. But it can be daunting when you are faced with an...

Of Metrics and Middleware

The topic of how the services in Weave Cloud are instrumented came up in two discussions this week, so I thought I’d write a short blog post explaining all the moving parts. Instrumenting Requests For all services in Weave Cloud, we...

Prometheus, ConfigMaps and Continuous Deployment

This is the story of how we manage our Prometheus config to avoid restarting Prometheus too often, losing all our history. It is a short, up to date write up of a talk I gave at the first London Prometheus meetup. In the beginning there...

Weave Cloud Outage 28th Feb Post-Mortem

Weave Cloud is our service for deploying, exploring, and monitoring microservice-based applications. At 17:37 GMT on 28th Feb, Weave Cloud suffered an outage for over 4 hours. The expectation of availablilty on a service like Weave Cloud...

Running Consul on Kubernetes and Monitoring it With Prometheus

We run a couple of replicated Consul services for Weave Cloud: one acts as a coordinator for the distributed WebSocket router backing Scope’s terminals feature, and the other for storing the state of Cortex’s consistent hash ring. This...

Aggregating Pod resource (CPU, memory) usage by arbitrary labels with Prometheus

How would you answer the questions like “how much CPU is my service consuming?” using Prometheus and Kubernetes? In this quick post, I’ll show you how… First we need to think about where to get the information from. cAdvisor (from Google)...

Turtles all the way down: HTTP over gRPC

I recently introduced a new mechanism for AuthFE (out authenticating reverse-proxy frontend for Weave Cloud) to forward requests to downstream services: HTTP over gRPC. This blog post covers why on earth I did this, and why you might want...

Dashboards as Code

If you use Prometheus, then you probably use Grafana. At Weave, we have Grafana dashboards for all of our microservices. When we want to understand our system, our Grafana dashboards are the first things we look at. To make the most out of...

Weave Mesh in Prometheus Alertmanager

tl;dr: Weave Mesh, an open-source gossip and CRDT communications library, gave the Prometheus Alertmanager exactly what it needed to be reliable and highly-available. As a former SoundCloud engineer, I’ve had the privilege of using...

Monitoring Your Kubernetes Infrastructure with Prometheus

In the last post in our series “Prometheus and Kubernetes”, Tom talked about how we use Prometheus to monitor the applications and services we deploy to Kubernetes. In this post, I want to talk about how we use Prometheus to monitor our...

Prometheus and Kubernetes: Monitoring Your Applications

In this post we discuss how to configure Prometheus to monitor your Kubernetes applications and services, and some best practices that we have developed around instrumenting your applications in a consistent fashion based on our experience...

Prometheus and Kubernetes: Deploying

In this post I discuss the various options and tradeoffs we encountered at Weaveworks when deploying Prometheus with Kubernetes to monitor Weave Cloud. This is the second post in our series on Prometheus and Kubernetes, the first being...

Prometheus and Kubernetes: A Perfect Match

At the inaugural Prometheus London meetup, I gave a talk about how Weaveworks uses Prometheus to monitor Weave Cloud, which runs on a Kubernetes cluster in AWS.  In this series of blog posts I’ll expand on some of the practices we’ve...