Recap: High-performance Linux Monitoring with eBPF
Recap of Alfonso Acosta’s talk High-performance Linux Monitoring with eBPF (Extended Berkeley Packet Filter). Learn how it’s used in Weave Scope.
On Saturday, March 25, I gave a talk for Source{d} on high-performance Linux monitoring with eBPF (Extended Berkeley Packet Filter) in their Madrid office. Here is a recap in case you missed it.
An overview of eBPF
To understand eBPF, you should first look at its ancestor: BPF (Berkeley Packet Filter). BPF is a high-performance user-level packet capture architecture. It was first introduced in 1992 by Berkeley Software Distribution (BSD). Hence how it got Berkeley in its name.Later, it was ported to other Unix systems like Linux. BPF has the goal of capturing or discarding packets (filtering) with the lowest possible overhead. Popular traffic analysis tools tcpdump or Wireshark use BPF to decide what packets should be captured. Packet filtering is carried out by an in-kernel virtual machine, executing a small program on every packet. The program's return value indicates the filtering result.
eBPF is an extension of BPF exclusive to Linux (it is included in the Linux Kernel mainline from version 3.18). It has a richer instruction set than BPF and expands its use cases beyond packet filtering. In my talk, I explained how eBPF can be used for dynamic tracing.
Not limited to networking filters
In addition to filtering packets, eBPF programs can be attached and run anywhere in the Linux Kernel through kprobes. This allows inspecting the execution of the kernel and userspace programs and opens the door to high-performance Linux monitoring without running extra kernel modules.
Potential kernel crashes caused by eBPF are prevented through static analysis. The kernel runs a verifier on every eBPF program before allowing its execution.
Without eBPF, kprobes must be injected in kernel modules, which is unsafe, dependent on the architecture, and exceptionally fragile. Though still sensitive to kernel-API changes, after using eBPF, the kprobes are injected from the userspace, safe, and architecturally independent.
Using eBPF for dynamic tracing is very powerful but equally terse to use. Development is becoming easier but still isn't ready for the faint-hearted. eBPF programs can now be coded in C thanks to an LLVM backend. The BPF Compiler Collection (BCC), simplifies development even further, but it comes with runtime dependencies (Python, kernel headers and LLVM) which were not acceptable for us.
In my talk I show how we used eBPF to inspect TCP connections in real-time from Golang, without any dependencies on LLVM nor kernel headers. To guarantee compatibility across kernel versions, the eBPF tracker uses offset guessing.
Weave Scope
So how does Weave Scope fit into all of this? An accurate, high-performance eBPF connection tracker is now available for use with Weave Scope. It is not yet enabled by default, so make sure to run Scope with --probe.ebpf.connections=true to try it out. Read how to get started here.
To watch my full talk from March 25, check out the video below.
