Extending Weave Scope With Custom Plugins
We show the essential information needed to create a plugin written in Go and point you to the resources you’ll need to do it yourself. The result should be a working plugin that you’ve built yourself.
We welcome back the Kinvolk team for a second guest post on extending Weave Scope with custom plugins. This is a continuation of their first topic "Testing Distributed Systems With Weave Scope’s Traffic Control Plugin”.
Weave Scope is a tool for monitoring and interacting with containers, processes and hosts. It is designed to be simple to install, use and extend. It’s extending Weave Scope that will be the topic of this post.
Specifically, we’ll show the essential information needed to create a plugin written in Go and point you to the resources you’ll need to do it yourself. The result should be a working plugin that you’ve built yourself. But there’s more to plugins than just running them locally. We’ll also walk you through the process of making the plugin deployable via Docker and Kubernetes.
And since you’ve gone through all the effort, you may want to share your plugin with others. You could contribute the code to the Weaveworks Scope Plugins repositories. We’ll point you there and give you some tips on how to get it merged.
So, let’s get going!
Motivation
Weave Scope provides a host of generally useful information (process info, connections, etc.) about the process, containers and hosts. But it’s likely there is particular information you’d like to make easily available in the Scope interface that isn’t exposed by default. Let’s assume, for the sake of this post, what we want to display is a host’s namespace information in the Scope side panel like this.
Along with the namespace table from our plugin, Scope also provides a helpful list of registered plugins.
About namespaces?
Namespaces are a Linux kernel feature that, for example, provide the isolation for containers. From the [Namespaces](http://man7.org/linux/man-pages/man7/namespaces.7.html) man page, “A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource.”
We can conveniently get namespace information with the [lsns](http://man7.org/linux/man-pages/man8/lsns.8.html) command which is usually found in the the util-linux package. Here is the output on my desktop system.
NS TYPE NPROCS PID USER COMMAND 4026531836 pid 61 2191 chris /usr/lib/systemd/systemd --user 4026531837 user 61 2191 chris /usr/lib/systemd/systemd --user 4026531838 uts 70 2191 chris /usr/lib/systemd/systemd --user 4026531839 ipc 70 2191 chris /usr/lib/systemd/systemd --user 4026531840 mnt 70 2191 chris /usr/lib/systemd/systemd --user 4026531969 net 61 2191 chris /usr/lib/systemd/systemd --user 4026532449 pid 1 14925 chris /opt/google/chrome/nacl_helper 4026532451 pid 2 14924 chris /opt/google/chrome/chrome --type=zygote 4026532453 net 8 14924 chris /opt/google/chrome/chrome --type=zygote 4026532561 net 1 14925 chris /opt/google/chrome/nacl_helper 4026532655 user 1 14925 chris /opt/google/chrome/nacl_helper 4026532656 user 8 14924 chris /opt/google/chrome/chrome --type=zygote 4026532658 pid 1 15741 chris /opt/google/chrome/chrome --type=renderer --field-trial-handle=1 --primordial-pipe-token=0C541509616056AEDBD5D23ADC02744D --lang=en-US -- 4026532659 pid 1 15068 chris /opt/google/chrome/chrome --type=renderer --field-trial-handle=1 --primordial-pipe-token=7E5A563C101995C656056EC30A0D6074 --lang=en-US -- 4026532661 pid 1 15149 chris /opt/google/chrome/chrome --type=renderer --field-trial-handle=1 --primordial-pipe-token=8017B0E65239D4D41F125D830B9A6D87 --lang=en-US -- 4026532673 pid 1 20689 chris /opt/google/chrome/chrome --type=renderer --field-trial-handle=1 --primordial-pipe-token=DE4A5DC80D6E8664BD72EA0E5BA88A97 --lang=en-US -- 4026532864 pid 1 18669 chris /opt/google/chrome/chrome --type=renderer --field-trial-handle=1 --primordial-pipe-token=5EB316A77D93DECF6ECFC43326495925 --lang=en-US --
You can see here that we’ve got different types of namespaces to isolate different resources: network, mount, pid, user, uts, and ipc.
The lsns command also supports outputting as JSON via the --json flag, which is what you’ll probably want to use since it’ll be easier to capture in Go structs.
Scope Plugin essentials
There are a couple essential requirements to make a Scope plugin work; it needs to register itself and it needs to send reports.
Registering the plugin
Registering is done by simply placing a unix socket in the /var/run/scope/plugins directory. Here’s some example code from the iowait plugin that does just that.
func setupSocket(socketPath string) (net.Listener, error) {
os.RemoveAll(filepath.Dir(socketPath))
if err := os.MkdirAll(filepath.Dir(socketPath), 0700); err != nil {
return nil, fmt.Errorf("failed to create directory %q: %v", filepath.Dir(socketPath), err)
}
listener, err := net.Listen("unix", socketPath)
if err != nil {
return nil, fmt.Errorf("failed to listen on %q: %v", socketPath, err)
}
log.Printf("Listening on: unix://%s", socketPath)
return listener, nil
}
We just need to listen on that socket and send reports.
http.HandleFunc("/report", plugin.Report)
Now let’s take a look at the report the handler function will return.
Sending reports
The data a plugin sends to Scope is referred to as a report, and must be properly formatted JSON. There are a number of different ways data can be displayed in Scope. The simplest is the property list. You can also show data graphically using metrics. Additionally, one can add interactions in the form of controls. But our plugin doesn’t need any of those. Instead it uses the multi-column table. How to define the desired display can be found in the report data structures documentation.
Below you’ll find an excerpt from the JSON we want to generate to add tabular namespace information to each host node.
"Host": {
"nodes": {
"myhostname;<host>": {
"latest": {
"host-namespaces-table-4026531836___host-namespaces-command": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "/usr/lib/systemd/systemd --switched-root --system --deserialize 23"
},
"host-namespaces-table-4026531836___host-namespaces-nprocs": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "294"
},
"host-namespaces-table-4026531836___host-namespaces-ns": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "4026531836"
},
"host-namespaces-table-4026531836___host-namespaces-pid": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "1"
},
"host-namespaces-table-4026531836___host-namespaces-type": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "pid"
},
"host-namespaces-table-4026531836___host-namespaces-user": {
"timestamp": "2017-05-08T04:54:01.688417869+02:00",
"value": "root"
},
"host-namespaces-table-4026531837___host-namespaces-command": {
"timestamp": "2017-05-08T04:54:01.688435798+02:00",
"value": "/usr/lib/systemd/systemd --switched-root --system --deserialize 23"
},
...
"table_templates": {
"host-namespaces-table-": {
"id": "host-namespaces-table-",
"label": "Host Namespaces",
"prefix": "host-namespaces-table-",
"type": "multicolumn-table",
"columns": [
{
"id": "host-namespaces-ns",
"label": "NS",
"dataType": ""
},
{
"id": "host-namespaces-type",
"label": "Type",
"dataType": ""
},
{
"id": "host-namespaces-nprocs",
"label": "# Procs",
"dataType": ""
},
{
"id": "host-namespaces-pid",
"label": "Pid",
"dataType": ""
},
{
"id": "host-namespaces-user",
"label": "User",
"dataType": ""
},
{
"id": "host-namespaces-command",
"label": "Command",
"dataType": ""
}
]
}
}
},
"Plugins": [
{
"id": "host-namespaces",
"label": "Host namespaces",
"description": "Display namespaces on host",
"interfaces": [
"reporter"
],
"api_version": "1"
}
]
The first line of this file indicates that we’re dealing with a Host topology. Other topologies are Container, Process, Pods, etc.
But let’s start by looking at the last section, Plugins. As the Plugins section’s documentation states, this is where we place the plugin metadata. It’s generally static but, as Scope doesn’t store state, you’ll need to include this with each report.
Above this is the section that describes what kind of elements (metadata_templates, table_templates, metric_templates, controls) we want to place into the Scope UI. In our case, we’re using the table_templates. Below you can see how each of these elements is displayed.
Table templates
A table template comes in two forms. If you don’t specify the “type” field, it’s assumed you want a property list, basically a list of label-value pairs. But we want to insert the namespace information as more traditional tabular data, in Scope terms a multi-column table. Thus, to define the table we want, we need to set the “type” field to multicolumn-table. Following the table element’s metadata, we provide a list of columns; id, label and type. If you need more help crafting your table, see the “Multicolumn table” section of the plugin docs. Now that we’ve defined the table, let’s look at how we get the data we want displayed into a format that Scope understands.
Report data
The real meat of the report is the data we want to display. Let’s take a look at that now, specifically what format we need to provide to get you namespace information into a table. Different elements require different formatting. This is described in the Report Data structure section of the plugin documentation.
The data for multicolumn tables are like all other elements in that it is placed in the latest section in the topology (Host) under the node(myhostname; ) with which the data is associated. Important to remember with respect to multicolumn-table is the format; “prefix{unique row id}___column-id” ⸻ that's 3 underscores. For our namespace plugin, the namespace id is a natural choice for the unique row identifier.
Putting this into code
You should now have all the tools and resources you need to write your own plugin. So you don’t get stuck on formatting the output of “lsns --json” here’s some code.
type NamespaceInfo struct {
NS string `json:"ns"`
Type string `json:"type"`
NProcs string `json:"nprocs"`
PID string `json:"pid"`
User string `json:"user"`
Command string `json:"command"`
}
type namespaces struct {
Namespaces []NamespaceInfo `json:"namespaces"`
}
func getNamespaces() (*namespaces, error) {
lsnsJson, err := lsns()
if err != nil {
return nil, err
}
var ns namespaces
if err := json.Unmarshal(lsnsJson, &ns); err != nil {
return nil, err
}
return &ns, nil
}
func lsns() ([]byte, error) {
out, err := exec.Command("lsns", "--json").Output()
if err != nil {
return nil, fmt.Errorf("lsns: %v", err)
}
return out, nil
}
So, go ahead, open your editor and put together the rest of the code. We’ll wait for you. ;)
Running a plugin
In order to test our plugin we need to build and run it. In Go, this is easy.
go build main.go sudo ./main
If you have Weave Scope running and select the host you’re running this on, you should see the plugin registered and a namespace table as depicted at the start of the post.
Deploying plugins
Generally, we don’t run our plugins from the command line. We want them to be deployed just as our applications are. So let’s create a Dockerfile for this.
FROM alpine:3.4 RUN apk add --no-cache util-linux COPY main /usr/bin/scope-namespace ENTRYPOINT ["/usr/bin/scope-namespace"]
Using Alpine Linux as a base, we install the util-linux package which includes the lsns tool. We then copy our plugin executable in and set it to start with the container. This plugin should gather information about hosts. Thus, it needs to be run with appropriate privileges and have access to the host’s namespaces. The `docker run` command should look something like this.
sudo docker run -it --net=host --pid=host --uts=host --ipc=host --privileged -v /var/run/scope/plugins:/var/run/scope/plugins <image id>
For Kubernetes, you’ll just need to create a yaml deployment file. The IOWait plugins deployment file is a good starting point. Notice that it’s of type DaemonSet so that it runs on all nodes. Also of note is that, similar to the `docker run` command, we specify that it should run with privileges and be in the host’s namespaces.
Conclusion
You should have all the tools you need to get started writing, running and deploying your own Scope plugins. We encourage you to look at the Scope plugin repositories and consider submitting the plugins you create.
Thank you for reading our blog. We build Weave Cloud, which is a hosted add-on to your clusters. It helps you iterate faster on microservices with continuous delivery, visualization & debugging, and Prometheus monitoring to improve observability.
Try it out, join our online user group for free talks & trainings, and come and hang out with us on Slack.
