Weaveworks Cortex - the newest member of the CNCF Sandbox
Cortex, our scalable Prometheus monitoring system, has been accepted as a Sandbox project by the Cloud Native Computing Foundation. Cortex is an open source project that we created to provide storage of Prometheus metrics for Weave Cloud. It's used by teams that are running large Prometheus environments, providing metrics for complex Kubernetes environments.
We're excited to announce that Cortex, our scalable Prometheus monitoring system, has been accepted as a Sandbox project by the Cloud Native Computing Foundation (CNCF). Cortex is an open source project that we created to provide storage of Prometheus metrics for Weave Cloud. It's used by teams that are running large Prometheus environments, providing metrics for complex Kubernetes environments. As advocates for open source communities we're looking forward to collaborating on Cortex's growth and development as part of the CNCF.
"The community of enterprise users and service providers around Cortex is validation of the need for a truly scalable Prometheus time series database," said Alexis Richardson, "With entry into the CNCF Sandbox we think this can only grow and fit many more use cases. Cortex scales using Kubernetes and we hope the project will follow a similar path - becoming the best way to store and scale container metrics everywhere”.
Why did we build Cortex?
In complex Kubernetes environments observability is critical for development and operations teams. Prometheus has become the defacto standard for metrics because it has several key strengths:
- Prometheus pulls metrics and auto-discovery
- the use of labels for arbitrary metrics
- the ability to co-deploy Kubernetes pods and Prometheus exporters
When the Weaveworks engineering team started adding monitoring to Weave Cloud we knew that Prometheus was the right choice. But, to build the SaaS we also wanted to store the metrics for a lot of users in a single system and store large amounts of time series data for long periods of time. That’s why we started developing Cortex.
For a more in depth background story - head over to our latest Medium post by Jonathan "What is Cortex".
What is Cortex?
Cortex is an open source tool that provides horizontally scalable, multi-tenant, long term storage for Prometheus metrics when used as a remote write destination, and a horizontally scalable, Prometheus-compatible query API.
The key capabilities are:
Prometheus Compatible
As Cortex was designed for our Weave Cloud SaaS the best way to think of it is as Prometheus-as-a-service. It allows you to provision on-demand “Prometheus Instances” with a Prometheus compatible API and query language - PromQL. PromQL is a concise and expressive query language, allowing for powerful insight into your application's behavior. Due to Prometheus’ dimensional data model, you can slice, dice and drill down quickly into problems - enabling you to rapidly analyse and resolve issues.
Horizontally Scalable and Multitenant
Cortex is designed to be horizontally scalable which is needed for large infrastructure deployments. It scales out in a single system that can handle hundreds of thousands of metrics from many systems. This contrasts with standard Prometheus where dealing with extensive metrics topologies is handled through federation which means running many Prometheus instances.
Cortex’s tenancy model is one cohesive system, not a node per client approach. Tenancy is encoded throughout the architecture and into the data storage layer. This means it supports users running large or multiple Kubernetes installations that require data separation. For example, in Weave Cloud multiple teams can attach their clusters and store their own data without intermingling it with other teams.
Long Term Storage and High Availability
Prometheus' local storage model limits the possible data retention and durability to that of a single machine. Cortex's use of cloud-native storage such as Amazon DynamoDB and S3 and on-premise NoSQL data stores, allow for virtually infinite data retention.
Built to support cloud native architectures that run with Kubernetes, Cortex is built on a highly available architecture.
Cortex in the CNCF
Cortex is part of our cloud native mission at Weaveworks:
“At Weaveworks we’re putting the developer experience at the center of everything we do. With Cortex we’re leveraging Prometheus’ powerful metrics to bring together data from many sources into a single observability system. We’re thrilled to have Cortex in the CNCF, a home that will foster collaboration with open source contributors from around the world.” says Matthias Radestock, CTO of Weaveworks.
Today Cortex users and contributors include Grafana Labs, FreshTracks (part of CA), EA and others.
“At FreshTracks.io we believe in bringing the right data to the right people at the right time. Cortex is at the heart of our offering providing multi-cluster cluster and multi-cloud visibility to developers and operators. The Cortex entry to the CNCF is exciting as it will provide the visibility and governance needed to ensure the ecosystem continues healthy growth.” says Bob Cotton, co-founder FreshTracks.io.
As a Sandbox project in the CNCF the aim is for the Cortex community to develop further. If you’re interested in developing a scalable Prometheus storage system then come and join the Cortex community!
And please join our community and help us grow the project:
