The Long Tail: Tools to Investigate High Long Tail Latency

Figuring out what's causing slow requests or ops can be tricky. Tom Wilkie shares the tools he used to investigate high long tail latency.

Kubernetes Rolling Upgrades One by One

Have a semi-stateful service running on Kubernetes and want to do rolling upgrades? Tom Wilkie outlines how to make deployments upgrade one by one.

The RED Method: key metrics for microservices architecture

Tom Wilkie shares Weaveworks monitoring philosophy and the three most important metrics to use in your microservices architecture.

WebSockets Are Evil and Must Die

In previous blog posts I’ve mention my dislike for WebSockets. In this quick blog post I attempt to explain why. Unfortunately I think this raises more questions than it answers!Be aware, these opinions are my own.It’s the monitoring,...

Prometheus UI: DIY with David Kaltschmidt from KubeCon Europe 2017

At KubeCon Europe 2017, Weaveworks Director of Software Engineering, UI/UX, David Kaltschmidt gave a presentation titled “Grafana is Not Enough: DIY User Interfaces for Prometheus.” In this talk, he discussed the Prometheus UI.What’s...

Loki: Distributed Tracing Zipkin/Prometheus Mashup with Tom Wilkie

Weaveworks Director of Software Engineering Tom Wilkie was busy at CloudNativeCon + KubeCon Europe 2017. Between showing off Cubienetes and presenting browser metrics in Prometheus, he found time to discuss one of his personal projects....

Weave Cloud UI Alerts and Metrics

After a particularly egregious outage was reported by a Weave Cloud user, we decided it would be a good idea to add some alerting to our front-end codebase. Alerting and metrics for Weave Cloud are handled using Prometheus, however, which...

Introduction to Prometheus Monitoring

Prometheus monitoring Prometheus is open source monitoring software that treats time series data as a data source for generating precise alerts. Now part of the Cloud Native Computing Foundation, the project began at SoundCloud in 2012. As...

Getting Browser Metrics into Prometheus: Tom Wilkie KubeCon Recap

During KubeCon, our Director of Software Engineering Tom Wilkie gave a presentation called “Behind the Iron Curtain: Getting Metrics From the Browser into Prometheus.”The ideaPreviously, Prometheus has been pigeon-holed as a monitoring...

PromQL queries for the rest of us

Prometheus comes with its own query language called PromQL. It’s very powerful and easily allows you to filter with the multi-dimensional time-series labels that make Prometheus so great. But it can be daunting when you are faced with an...

Of Metrics and Middleware

The topic of how the services in Weave Cloud are instrumented came up in two discussions this week, so I thought I’d write a short blog post explaining all the moving parts.Instrumenting RequestsFor all services in Weave Cloud, we measure...

Prometheus, ConfigMaps and Continuous Deployment

This is the story of how we manage our Prometheus config to avoid restarting Prometheus too often, losing all our history. It is a short, up to date write up of a talk I gave at the first London Prometheus meetup. In the beginning there...

Running Consul on Kubernetes and Monitoring it With Prometheus

We run a couple of replicated Consul services for Weave Cloud: one acts as a coordinator for the distributed WebSocket router backing Scope’s terminals feature, and the other for storing the state of Cortex’s consistent hash ring. This...

Aggregating Pod resource (CPU, memory) usage by arbitrary labels with Prometheus

How would you answer the questions like “how much CPU is my service consuming?” using Prometheus and Kubernetes? In this quick post, I’ll show you how…First we need to think about where to get the information from. cAdvisor (from Google)...

Turtles all the way down: HTTP over gRPC

I recently introduced a new mechanism for AuthFE (out authenticating reverse-proxy frontend for Weave Cloud) to forward requests to downstream services: HTTP over gRPC. This blog post covers why on earth I did this, and why you might want...

Cloud Provider Options for a Container-Friendly Environment

The advent Docker and containers, has simplified packaging, deploying and supporting applications. However, with this advancement come challenges, as you also need a robust environment in which to deploy the containers. And that requires...

Dashboards as Code

If you use Prometheus, then you probably use Grafana. At Weave, we have Grafana dashboards for all of our microservices. When we want to understand our system, our Grafana dashboards are the first things we look at. To make the most out of...

Join Weaveworks at KubeCon Seattle + CloudNativeCon

KubeCon Seattle is Almost Here KubeCon Seattle is just around the corner and we hope to see you there Nov. 8-9. If you’re not familiar with KubeCon Seattle, the two-day event brings together thousands of Kubernetes technologists to hear...

Weave Mesh in Prometheus Alertmanager

tl;dr: Weave Mesh, an open-source gossip and CRDT communications library, gave the Prometheus Alertmanager exactly what it needed to be reliable and highly-available.As a former SoundCloud engineer, I’ve had the privilege of using...

Monitoring Your Kubernetes Infrastructure with Prometheus

In the last post in our series “Prometheus and Kubernetes”, Tom talked about how we use Prometheus to monitor the applications and services we deploy to Kubernetes. In this post, I want to talk about how we use Prometheus to monitor our...