Write your first Weave Scope Plugin

Version 0.14 of Weave Scope came with an exciting new feature: plugins. If you’re visualizing and monitoring your microservices with Weave Scope, plugins provide a way to enhance and enrich the view with your own custom data. This data...

Version 0.14 of Weave Scope came with an exciting new feature: plugins. If you’re visualizing and monitoring your microservices with Weave Scope, plugins provide a way to enhance and enrich the view with your own custom data. This data could be specific to your application, like how many visitors are being served by a given container, or it could be specific to your infrastructure, like the AWS instance type of a host. There are several starter plugins at: https://github.com/weaveworks-plugins, with examples of monitoring HTTP requests, and controlling network traffic to introduce latency when testing. You can find additional documentation for plugins in the Scope repository.

We’ll walk you through how to build a simple plugin which will display the number of mounted volumes for each container. Even though it is basic, this example will cover all you need to know to add nodes and metadata into Scope’s visualization.

Weave Scope Plugins

Scope plugins are simple to implement, and can be written in any language. Each scope plugin must respond to HTTP requests on a unix socket. Scope will make periodic requests to the plugin for information. When a plugin exits, it should remove the unix socket. Negotiating the plugin’s ID, its capabilities, and reporting data, are all done through HTTP requests to the socket. Scope-to-plugin communication centers around a JSON data-structure called a “report.” It is the same data-structure that scope probes send to the app. If you have a running scope app, you can see an example of a report by visiting “/api/report”, or clicking the “</>” icon in the bottom right corner. There are a couple differences between reports from probes and reports from plugins. First and foremost, is the “Plugins” section, which plugins must use to report the information about themselves to the Scope App. Secondly, is that reports from plugins do not have to include all data about the system. Plugin reports can be just the minimum information the plugin would like to merge into the app. More in-depth information is available in the readme.

1 – Plugin Basics

The following instructions explain how to build a basic Scope plugin. It will have an HTTP server responding to Scope’s requests for data, and on each request it will contact Docker, fetch the number of volumes for each container, and reformat that for Scope.

You’ll start by setting up a basic HTTP server. Using python as an example, you can set up an HTTP server easily by just importing BaseHTTPServer. However, because it is a unix socket, you have to specify the listening path a bit differently.

<code>
#!/usr/bin/env python

import BaseHTTPServer
import SocketServer
import errno
import os
import signal
import socket

PLUGIN_ID="volume-count"
PLUGIN_UNIX_SOCK = "/var/run/scope/plugins/" + PLUGIN_ID + ".sock"

class Handler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(self):
        # The logger requires a client_address, but unix sockets don't have
        # one, so we fake it.
        self.client_address = "-"

        # Send the headers
        self.send_response(200)
        self.end_headers()

        # Send the body
        self.wfile.write(b"foo")

def mkdir_p(path):
    try:
        os.makedirs(path)
    except OSError as exc:
        if exc.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else:
            raise

def main():
    # Ensure the socket directory exists
    mkdir_p(os.path.dirname(PLUGIN_UNIX_SOCK))
    # Listen for connections on the unix socket
    server = SocketServer.UnixStreamServer(PLUGIN_UNIX_SOCK, Handler)
    server.serve_forever()

main()
</code>

To handle requests, implement a “BaseHTTPServer.BaseHTTPRequestHandler.” To test it is working, use curl. Specify the unix socket to connect to separately. At this point you should see:

<code>

$ curl --unix-socket /var/run/scope/plugins/volume-count.sock http:///report
foo

</code>

This is fine, and indicates our basic server is working.

2 – Cleanup

It is good practice to remove the unix socket when your plugin exits. If you leave it behind, Scope will keep requesting information from your plugin, even though it is no longer running. This won’t cause any practical issues, but will clutter the logs with “plugin failed” messages. There are a few instances for which cleanup is necessary. First, use the “signal” library to set up a signal handler. This will remove the socket when the plugin exits.

<code>
def delete_socket_file():
    if os.path.exists(PLUGIN_UNIX_SOCK):
        os.remove(PLUGIN_UNIX_SOCK)

def sig_handler(b, a):
    delete_socket_file()
    exit(0)

def main():
    signal.signal(signal.SIGTERM, sig_handler)
    signal.signal(signal.SIGINT, sig_handler)
    ...
</code>

Second, you will need to handle cleanup in the case of the HTTP server failing to start.

<code>
    # Listen for connections on the unix socket
    server = SocketServer.UnixStreamServer(PLUGIN_UNIX_SOCK, Handler)
    try:
        server.serve_forever()
    except:
        delete_socket_file()
        raise
</code>

Lastly, perform a cleanup when the plugin boots, just in case the previously running plugin couldn’t perform cleanup. Note, this last will mean that if you launch the plugin twice, the second one to start up will replace the first.

<code>
    # Ensure the socket directory exists
    mkdir_p(os.path.dirname(PLUGIN_UNIX_SOCK))
    # Remove existing socket in case it was left behind
    delete_socket_file()
</code>

3 – Registering the plugin with Scope

At this point you’re ready to start returning data to Scope. All plugins must report some basic information about themselves. This is done as part of the “Plugins” section of the response. You can find more information about the individual fields in the plugins documentation.

<code>
       # Generate our json body
       body = json.dumps({
            'Plugins': [
               {
                   'id': PLUGIN_ID,
                   'label': 'Volume Counts',
                   'description': 'Shows how many volumes each container has mounted',
                   'interfaces': ['reporter'],
                   'api_version': '1',
               }
            ]
       })

       # Send the headers
       self.send_response(200)
       self.send_header('Content-type', 'application/json')
       self.send_header('Content-length', len(body))
       self.end_headers()

       # Send the body
       self.wfile.write(body)
</code>

Now that the plugin is set up and registering with Scope, you can have it add some basic data to the UI. The report data-structure is divided up into several topologies. Each topology has information about a set of nodes. Those topologies are aggregated and filtered, to produce the various views in Scope. The topologies available are: Host, Container, ContainerImage, Pod, Service, Deployment, ReplicaSet, and Endpoint. Each topology contains the set of nodes, as well as some information on how to render the metadata and metrics of the nodes.

Note that the data returned by a plugin must not be greater than 50 MB. Also, to allow scope time to aggregate the data within its time limits each plugin must respond in less than 500ms. If a plugin takes longer than that to respond the data will be ignored.

4 – Adding our first metadata

Next you’ll want to add some container nodes into that topology.

<code>
        # Generate our json body
        body = json.dumps({
            'Plugins': [
                ...
            ],
            'Container': {
                'nodes': {
                    'abcd1234;': {
                        'latest': {
                            'volume_count': {
                                'timestamp': timestamp,
                                'value': '1',
                            }
                        }
                    }
                },
                # Templates tell the UI how to render this field.
                'metadata_templates': {
                    'volume_count': {
                        # Key where this data can be found.
                        'id': "volume_count",
                        # Human-friendly field name
                        'label': "# Volumes",
                        # Look up the 'id' in the latest object.
                        'from': "latest",
                        # Priorities over 10 are hidden, lower is earlier in the list.
                        'priority': 0.1,
                    },
                },
            },
        })
</code>

Each topology has a unique format for its node IDs. In this example, for the container topology, the format is the full, long container ID, followed by “;”. More examples of node IDs can be found in the plugins documentation.

Nodes from multiple sources, for example from different probes or plugins, are merged based on their Topology and node ID. In order for the data to be merged onto existing containers, the Node ID must match. If your plugin returns a node ID which doesn’t match any others, the new node will be created.

In order to render the new metadata from your plugin, you must provide a “metadata_templates” field with information about how to render the given field. Only fields with a “metadata_templates”, “metric_templates”, or “table_templates” entry will be rendered into the UI.

If you view the running scope UI, and launch the plugin, you will see that a blank new container “abcd1234” has been created, and it has new metadata associated with it.

5 – Fetching and formatting data from Docker

Now you’re ready to get some real data from docker to finish your plugin. Since this plugin is being written in python, you can use the docker-py API client. Use a pip install to make that available, then fetch the data by listing all containers, and examining the Mounts field for each.

<code>

    from docker import Client

    # List all containers, with the count of their volumes
    def container_volume_counts():
    containers = {}
    cli = Client(base_url=DOCKER_SOCK, version='auto')
    for c in cli.containers(all=True):
        containers[c['Id']] = len(c['Mounts'])
    return containers

</code>

Next, go through all the results, and convert each count to the appropriate container node in Scope. Add the current timestamp onto each value, so that they get merged correctly.

<code>
        # Get current timestamp in RFC3339
        timestamp = datetime.datetime.utcnow()
        timestamp = timestamp.isoformat('T') + 'Z'

        # Fetch and convert data to scope data model
        nodes = {}
        for container_id, volume_count in container_volume_counts().iteritems():
            nodes["%s;" % (container_id)] = {
                'latest': {
                    'volume_count': {
                        'timestamp': timestamp,
                        'value': str(volume_count),
                    }
                }
            }
</code>

Success

When you run your plugin on the same host as a Scope probe, we will begin seeing our plugin listed in the UI. It will be listed in the bottom right, next to the “Pause” button. Our metadata from Docker, will show up as info entries on the node details panel.