We’re very happy today to give the spotlight to Kai Davenport, who has been advancing the state of the art of Docker extensions, and weave, with his project powerstrip-weave.

–Michael

Powerstrip-Weave: A Docker Networking Extension

In this post I will discuss the inner workings of powerstrip-weave – a way of composing weave with other docker extensions.

The tool that enables this composition is powerstrip, a project by ClusterHQ, the makers of docker data management tool Flocker.

Docker Extensions

The discussion surrounding docker and how to extend it has created much buzz (and some consternation) in recent months.

There is recognition that solutions to the following problems are needed in order to run a multi-host docker cluster:

  • volume management
  • networking
  • service discovery
  • scheduling

The phrase batteries included but removable hints at the idea that an ecosystem of extensions, authored by members of the community, is what we should be aiming for.

Powerstrip

This leads us to powerstrip – a tool for prototyping docker extensions.

Powerstrip allows a single docker host to trigger multiple “adapters” that can hook into ANY of the standard docker API calls.

An adapter can hook in before a request arrives (a “pre-hook”) and after a response is received (a “post-hook”). This allows an adapter to modify or react to any part of a container’s life cycle.

powerstrip is a docker proxy

powerstrip presents a docker proxy meaning the vanilla docker client (and by extension a whole range of orchestration tools) can communicate with it.

It does this by listening to tcp://0.0.0.0:2375 leaving the real docker server to listen to unix:///var/run/docker.sock.

powerstrip intercepts and can modify HTTP requests (like POST /containers/create) both to and from the docker server.

pre-hooks

pre-hooks allow an adapter to modify docker requests before they reach the docker server. This is useful because an adapter can (amongst other things):

  • inject ENV variables or add volumes to a container before it is created
  • contact the docker server to ask for information about the image a container is based on
  • delay the start of a container until some condition is met
post-hooks

post-hooks allow adapters to intercept docker responses before they are returned to the docker client. This is useful because an adapter can (amongst other things):

  • update etcd or consul with endpoint information once the container has started
  • log completed docker requests to an external service
  • trigger external services that require the id of a started container

Weave

weave currently “wraps” Docker. This means that when a call to weave run is made – it will internally call docker run and then attach an IP address to the container once it has started.

This can be seen by this diagram:

In this flow – the following events happen in the following order:

1. weave run

Firstly, weave run is passed the CIDR notation of an IP address followed by the arguments that are passed to the docker run command.

2. docker run

Secondly, the docker container is started (in detached mode) by passing any arguments that followed the weave IP address. The $CONTAINER_ID returned by docker is grabbed by weave.

3. weave attach

Finally, weave will give the container an IP address on the weave network.

The problem

The flow of events above means there is a small amount of time during which the container is not connected to the weave network (the time between docker run and weave attach), and that the weave-connected container cannot be run in the foreground.

These can cause problems for some containers and without changes to docker itself, they are difficult to solve.

powerstrip-weave

What if there was a way to run a container but tell it to WAIT until the weave network was connected before actually running?

Well it turns out that this is exactly what powerstrip-weave is designed to do. A “pre-hook” hijacking the entry-point of a container and a “post-hook” triggering weave attach means we can have weave containers waiting until they are connected before running.

Here’s how to try it out.

Set up

First we must get everything up and running – this includes:

NOTE – the following example presently works on ubuntu. RedHat and CoreOS will work when this issue has been solved (pull requests welcome!)

1. create config

The first step is to configure powerstrip with a “pre-hook” for /containers/create and a “post-hook” for /containers/*/start:

$ cat > ~/powerstrip-demo/adapters.yml <<EOF
endpoints:
  "POST /*/containers/create":
    pre: [weave]
  "POST /*/containers/*/start":
    post: [weave]
adapters:
  weave: http://weave/v1/extension
EOF
2. start powerstrip-weave

Now we start powerstrip weave:

$ docker run -d --name powerstrip-weave 
    --expose 80 
    -v /var/run/docker.sock:/var/run/docker.sock 
    -v /usr/bin/docker:/usr/bin/docker 
    binocarlos/powerstrip-weave launch
3. start powerstrip

Finally we start the powerstrip server itself:

$ docker run -d --name powerstrip 
  -v /var/run/docker.sock:/var/run/docker.sock 
  -v ~/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml 
  --link powerstrip-weave:weave 
  -p 2375:2375 
  clusterhq/powerstrip

Pull Image

To help the example run smoother – pull the example image:

$ docker pull binocarlos/powerstrip-weave-example:latest

Run Containers!

Now we can run a container using the standard docker client that will auto-assign a weave IP address and wait for it to be connected before starting the process:

$ DOCKER_HOST=tcp://127.0.0.1:2375 CID=$(docker run -e "WEAVE_CIDR=10.255.0.51/24" -d binocarlos/powerstrip-weave-example hello world)
$ docker logs $CID
$ docker rm $CID

Notice how we pass an environment variable WEAVE_CIDR to the container we are running. This tells powerstrip-weave the IP we want the container to have.

You should see that the output reports that the network took zero seconds to connect and that the ethwe interface has an IP address of 10.255.0.51.

How it works

The following is a breakdown of how powerstrip-weave pulls off this feat.

1. weavewait volume

On start up – powerstrip-weave will start a container called weavewait.

This is from the binocarlos/wait-for-weave image which is a golang program that will block until the weave network is connected. Once the weave network connects, it will execute whatever is passed as arguments as a command.

So for example – running this command:

$ wait-for-weave bash -c "echo hello"

Will output hello but only AFTER the weave network is connected.

2. Hijack POST /containers/create requests

There is a “pre-hook” that will do the following things for requests to /containers/create:

  • check the Environment variables for a WEAVE_CIDR value and skip if not present
  • load the original Entrypoint from docker based on the image used
  • prepends the Entrypoint (image or container) onto the Cmd
  • set the Entrypoint to /home/weavewait/wait-for-weave
  • append --volumes-from=weavewait to the HostConfig of the container

This has the effect of:

  • making the wait-for-weave binary available to the container
  • making the wait-for-weave binary run first when the container starts
  • ensuring that the original entry-point is preserved even if not supplied to the docker run command
  • running the entry-point after the weave network is connected

So for example – if our request to /containers/create was:

{
  "Image":"myappimage",
  "Env":[
    "WEAVE_CIDR=10.255.0.1/24"
  ],
  "Cmd":["--fruit", "apples"],
  "Entrypoint":null,
  "HostConfig":{
    "VolumesFrom":null
  }
}

And the JSON for /images/myappimage/json was:

{
  "Entrypoint":["bash", "/app/script.sh"]
}

Then powerstrip-weave would remap the container JSON into:

{
  "Image":"myappimage",
  "Env":[
    "WEAVE_CIDR=10.255.0.1/24"
  ],
  "Cmd":["bash", "/app/script.sh", "--fruit", "apples"],
  "Entrypoint":["/home/weavetools/wait-for-weave"],
  "HostConfig":{
    "VolumesFrom":["weavetools"]
  }
}
3. Intercept POST /containers/*/start requests

There is a “post-hook” that will do the following things to requests to /containers/*/start:

  • load the Environment variables from docker for the $CONTAINER_ID and grab the value of WEAVE_CIDR
  • run a weave attach $WEAVE_CIDR $CONTAINER_ID command

This has the effect of:

  • loading the intended IP address meant for the container
  • telling weave to attach the IP address using the $CONTAINER_ID

Combined

As soon as the weave attach command is run in the post-hook and the weave IP address has been assigned, the copy of wait-for-weave that is currently blocking in the container will notice the weave network and proceed to run the original entry-point.

The real thing to note is how a powerstrip adapter is combining two different stages of a docker run command (create & start) to achieve the intended outcome.

Conclusion

Hopefully this post has shown the power of powerstrip (pun intended).

Moving forward, if we all start developing a wide range of adapters then everyone can benefit from the freedom to choose how to compose their containers from lots of small solutions – batteries in all shapes and sizes.