Powerstrip-Weave
We’re very happy today to give the spotlight to Kai Davenport, who has been advancing the state of the art of Docker extensions, and weave, with his project powerstrip-weave. –Michael Powerstrip-Weave: A Docker Networking Extension In this...
We’re very happy today to give the spotlight to Kai Davenport, who has been advancing the state of the art of Docker extensions, and weave, with his project powerstrip-weave.
–Michael
Powerstrip-Weave: A Docker Networking Extension
In this post I will discuss the inner workings of powerstrip-weave – a way of composing weave with other docker extensions.
The tool that enables this composition is powerstrip, a project by ClusterHQ, the makers of docker data management tool Flocker.
Docker Extensions
The discussion surrounding docker and how to extend it has created much buzz (and some consternation) in recent months.
There is recognition that solutions to the following problems are needed in order to run a multi-host docker cluster:
- volume management
- networking
- service discovery
- scheduling
The phrase batteries included but removable hints at the idea that an ecosystem of extensions, authored by members of the community, is what we should be aiming for.
Powerstrip
This leads us to powerstrip – a tool for prototyping docker extensions.
Powerstrip allows a single docker host to trigger multiple “adapters” that can hook into ANY of the standard docker API calls.
An adapter can hook in before a request arrives (a “pre-hook”) and after a response is received (a “post-hook”). This allows an adapter to modify or react to any part of a container’s life cycle.
powerstrip is a docker proxy
powerstrip presents a docker proxy meaning the vanilla docker client (and by extension a whole range of orchestration tools) can communicate with it.
It does this by listening to tcp://0.0.0.0:2375 leaving the real docker server to listen to unix:///var/run/docker.sock.
powerstrip intercepts and can modify HTTP requests (like POST /containers/create) both to and from the docker server.
pre-hooks
pre-hooks allow an adapter to modify docker requests before they reach the docker server. This is useful because an adapter can (amongst other things):
- inject ENV variables or add volumes to a container before it is created
- contact the docker server to ask for information about the image a container is based on
- delay the start of a container until some condition is met
post-hooks
post-hooks allow adapters to intercept docker responses before they are returned to the docker client. This is useful because an adapter can (amongst other things):
- update etcd or consul with endpoint information once the container has started
- log completed docker requests to an external service
- trigger external services that require the id of a started container
Weave
weave currently “wraps” Docker. This means that when a call to weave run is made – it will internally call docker run and then attach an IP address to the container once it has started.
This can be seen by this diagram:
In this flow – the following events happen in the following order:
1. weave run
Firstly, weave run is passed the CIDR notation of an IP address followed by the arguments that are passed to the docker run command.
2. docker run
Secondly, the docker container is started (in detached mode) by passing any arguments that followed the weave IP address. The $CONTAINER_ID returned by docker is grabbed by weave.
3. weave attach
Finally, weave will give the container an IP address on the weave network.
The problem
The flow of events above means there is a small amount of time during which the container is not connected to the weave network (the time between docker run and weave attach), and that the weave-connected container cannot be run in the foreground.
These can cause problems for some containers and without changes to docker itself, they are difficult to solve.
powerstrip-weave
What if there was a way to run a container but tell it to WAIT until the weave network was connected before actually running?
Well it turns out that this is exactly what powerstrip-weave is designed to do. A “pre-hook” hijacking the entry-point of a container and a “post-hook” triggering weave attach means we can have weave containers waiting until they are connected before running.
Here’s how to try it out.
Set up
First we must get everything up and running – this includes:
NOTE – the following example presently works on ubuntu. RedHat and CoreOS will work when this issue has been solved (pull requests welcome!)
1. create config
The first step is to configure powerstrip with a “pre-hook” for /containers/create and a “post-hook” for /containers/*/start:
$ cat > ~/powerstrip-demo/adapters.yml <<EOF
endpoints:
"POST /*/containers/create":
pre: [weave]
"POST /*/containers/*/start":
post: [weave]
adapters:
weave: http://weave/v1/extension
EOF2. start powerstrip-weave
Now we start powerstrip weave:
$ docker run -d --name powerstrip-weave
--expose 80
-v /var/run/docker.sock:/var/run/docker.sock
-v /usr/bin/docker:/usr/bin/docker
binocarlos/powerstrip-weave launch3. start powerstrip
Finally we start the powerstrip server itself:
$ docker run -d --name powerstrip
-v /var/run/docker.sock:/var/run/docker.sock
-v ~/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml
--link powerstrip-weave:weave
-p 2375:2375
clusterhq/powerstripPull Image
To help the example run smoother – pull the example image:
$ docker pull binocarlos/powerstrip-weave-example:latestRun Containers!
Now we can run a container using the standard docker client that will auto-assign a weave IP address and wait for it to be connected before starting the process:
$ DOCKER_HOST=tcp://127.0.0.1:2375 CID=$(docker run -e "WEAVE_CIDR=10.255.0.51/24" -d binocarlos/powerstrip-weave-example hello world)
$ docker logs $CID
$ docker rm $CIDNotice how we pass an environment variable WEAVE_CIDR to the container we are running. This tells powerstrip-weave the IP we want the container to have.
You should see that the output reports that the network took zero seconds to connect and that the ethwe interface has an IP address of 10.255.0.51.
How it works
The following is a breakdown of how powerstrip-weave pulls off this feat.
1. weavewait volume
On start up – powerstrip-weave will start a container called weavewait.
This is from the binocarlos/wait-for-weave image which is a golang program that will block until the weave network is connected. Once the weave network connects, it will execute whatever is passed as arguments as a command.
So for example – running this command:
$ wait-for-weave bash -c "echo hello"Will output hello but only AFTER the weave network is connected.
2. Hijack POST /containers/create requests
There is a “pre-hook” that will do the following things for requests to /containers/create:
- check the Environment variables for a
WEAVE_CIDRvalue and skip if not present - load the original
Entrypointfrom docker based on the image used - prepends the
Entrypoint(image or container) onto theCmd - set the
Entrypointto/home/weavewait/wait-for-weave - append
--volumes-from=weavewaitto the HostConfig of the container
This has the effect of:
- making the
wait-for-weavebinary available to the container - making the
wait-for-weavebinary run first when the container starts - ensuring that the original entry-point is preserved even if not supplied to the docker run command
- running the entry-point after the weave network is connected
So for example – if our request to /containers/create was:
{
"Image":"myappimage",
"Env":[
"WEAVE_CIDR=10.255.0.1/24"
],
"Cmd":["--fruit", "apples"],
"Entrypoint":null,
"HostConfig":{
"VolumesFrom":null
}
}And the JSON for /images/myappimage/json was:
{
"Entrypoint":["bash", "/app/script.sh"]
}Then powerstrip-weave would remap the container JSON into:
{
"Image":"myappimage",
"Env":[
"WEAVE_CIDR=10.255.0.1/24"
],
"Cmd":["bash", "/app/script.sh", "--fruit", "apples"],
"Entrypoint":["/home/weavetools/wait-for-weave"],
"HostConfig":{
"VolumesFrom":["weavetools"]
}
}3. Intercept POST /containers/*/start requests
There is a “post-hook” that will do the following things to requests to /containers/*/start:
- load the Environment variables from docker for the $CONTAINER_ID and grab the value of
WEAVE_CIDR - run a
weave attach $WEAVE_CIDR $CONTAINER_IDcommand
This has the effect of:
- loading the intended IP address meant for the container
- telling weave to attach the IP address using the $CONTAINER_ID
Combined
As soon as the weave attach command is run in the post-hook and the weave IP address has been assigned, the copy of wait-for-weave that is currently blocking in the container will notice the weave network and proceed to run the original entry-point.
The real thing to note is how a powerstrip adapter is combining two different stages of a docker run command (create & start) to achieve the intended outcome.
Conclusion
Hopefully this post has shown the power of powerstrip (pun intended).
Moving forward, if we all start developing a wide range of adapters then everyone can benefit from the freedom to choose how to compose their containers from lots of small solutions – batteries in all shapes and sizes.
