Running Containers on Remote Docker Hosts with Weave 0.10

April 29, 2015

feature

implementation

In an earlier post we peeked under the covers of Weave 0.10 to discover how it interacts with remote Docker hosts. So, for example weave launch is translated into: docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v...

Docker Networking with No External Cluster Store: Announcing Weave 1.4 Plugin

How Weave built a cloud deployment for Scope using Kubernetes

Weave Scope gets your Docker containers under control

In an earlier post we peeked under the covers of Weave 0.10 to discover how it interacts with remote Docker hosts. So, for example weave launch is translated into:
 docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local launch 
Other weave commands are translated in the same way. Except weave run. Following the above pattern we would expect weave run 10.2.0.1/24 -ti ubuntu to turn into:
 docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local run 10.2.0.1/24 -ti ubuntu 
But that is not what happens…

Stuck between versions

The above translation of weave run does in fact work, but only if all the run options specified are supported by Docker 1.3.1. That is because the weaveexec image embeds a Docker client of that version. We chose such a relatively old version because we want Weave to run in places that may not have the latest Docker version installed. Docker daemons are backward compatible, so an old client can talk to a newer daemon. Hence our choice of Docker version in the weaveexec image means Weave will work with all Docker daemons >= 1.3.1. But the downside is that any interactions with Docker through weaveexec must use options that the embedded Docker 1.3.1 understands.

So how do we let users employ features of Docker versions > 1.3.1? This is most pertinent to weave run, which is the primary means of starting application containers attached to the Weave network. The options to docker run, which it invokes, have evolved significantly between Docker versions. So, for example, a user running Docker 1.6 may want to attach a container label – a new feature in 1.6 – with --label=... when invoking weave run. If weave run was executing via the weaveexec image as shown above, that would result in an error, since the Docker 1.3.1 client embedded in the image wouldn’t know how to parse the --label=... parameter.

In a way we are trying to satisify an impossible constraint: we want the weaveexec Docker version to be old, so it works with old daemons, and new, so it supports recently introduced docker run options.

2 for 1

The solution we implemented in the weave script is to translate weave run into a combination of docker run and weave attach. The former bypasses the weaveexec image, and hence is not constrained by that image’s Docker version.

Let’s see what an execution of weave run 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu actually looks like…
 $ sh -x weave run 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + set -e + SCRIPT_VERSION=0.10.0 + [ 0.10.0 = (unreleased version) ] + IMAGE_VERSION=0.10.0 + IMAGE_VERSION=0.10.0 + DOCKERHUB_USER=weaveworks + BASE_EXEC_IMAGE=weaveworks/weaveexec + EXEC_IMAGE=weaveworks/weaveexec:0.10.0 + PROCFS=/proc + [ run = run ] + shift 1 + check_docker_version + docker -v + sed -n -e s|^Docker version ([0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*).*|1|p + DOCKER_VERSION=1.6.0 + [ -z 1.6.0 ] + [ 1.6.0 = 1.3.0 ] + [ 10.2.0.1/24 = --with-dns ] + collect_cidr_args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + CIDR_ARGS= + CIDR_COUNT=0 + is_cidr 10.2.0.1/24 + echo 10.2.0.1/24 + grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$ + CIDR_ARGS= 10.2.0.1/24 + CIDR_COUNT=1 + shift 1 + is_cidr --label=com.foo.deployment=stage + echo --label=com.foo.deployment=stage + grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$ + [ 1 -eq 0 ] + shift 1 + docker run -d --label=com.foo.deployment=stage -ti ubuntu + CONTAINER=0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa + exec_remote attach 10.2.0.1/24 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa + docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local attach 10.2.0.1/24 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa + echo 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa + exit 0

There is a lot more going on here than in weave launch trace we saw earlier, though it is not nearly as complicated as it may first appear. The initial execution path, up to the setting of EXEC_IMAGE is the same as for weave launch. Next the script checks that we are not running Docker 1.3.1, which contains a nasty bug that prevents weave run from working. Then the script strips off all the weave specific parameters, such as the --with-dns flag and CIDR args (in this case, the 10.2.0.1/24 we supplied). It then invokes docker run -d to start the application container in the background, with all the non-weave arguments. The resulting container id is given to weave attach, together with the collected CIDRs. The weave attach is translated into an invocation of a container running the weaveexec image, exactly as we have seen previously. Finally, we echo the container id back to the user.

All pretty simple, really, and fairly easy to grasp from the code in the relevant section of the script:
 if [ "$1" = "run" ]; then shift 1 check_docker_version if [ "$1" = "--with-dns" ]; then shift 1 DNS_ARGS=$(exec_remote dns-args "$@") fi collect_cidr_args "$@" shift $CIDR_COUNT CONTAINER=$(docker run $DNS_ARGS -d "$@") exec_remote attach $CIDR_ARGS $CONTAINER echo $CONTAINER exit 0 ... 

But the astute reader will notice another complication in the above…

3 containers for 1 command

weave run with the --with-dns flag requires starting the application container with extra dns-related parameters to docker run. Figuring our what those should be is not completely trivial, and requires parsing of the supplied arguments, and determining the IP of the docker bridge. The script is delegating that task to the weaveexec container, invoking the equivalent of weave dns-args .... The output contains the dns-related parameters, which are spliced into the invocation of docker run.

Here’s what a trace of weave run --with-dns 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu looks like…
 $ sh -x weave run --with-dns 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + set -e + SCRIPT_VERSION=0.10.0 + [ 0.10.0 = (unreleased version) ] + IMAGE_VERSION=0.10.0 + IMAGE_VERSION=0.10.0 + DOCKERHUB_USER=weaveworks + BASE_EXEC_IMAGE=weaveworks/weaveexec + EXEC_IMAGE=weaveworks/weaveexec:0.10.0 + PROCFS=/proc + [ run = run ] + shift 1 + check_docker_version + docker -v + sed -n -e s|^Docker version ([0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*).*|1|p + DOCKER_VERSION=1.6.0 + [ -z 1.6.0 ] + [ 1.6.0 = 1.3.0 ] + [ --with-dns = --with-dns ] + shift 1 + exec_remote dns-args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local dns-args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + DNS_ARGS=--dns 172.17.42.1 --dns-search=. + collect_cidr_args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu + CIDR_ARGS= + CIDR_COUNT=0 + is_cidr 10.2.0.1/24 + echo 10.2.0.1/24 + grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$ + CIDR_ARGS= 10.2.0.1/24 + CIDR_COUNT=1 + shift 1 + is_cidr --label=com.foo.deployment=stage + echo --label=com.foo.deployment=stage + grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$ + [ 1 -eq 0 ] + shift 1 + docker run --dns 172.17.42.1 --dns-search=. -d --label=com.foo.deployment=stage -ti ubuntu + CONTAINER=f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9 + exec_remote attach 10.2.0.1/24 f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9 + docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local attach 10.2.0.1/24 f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9 + echo f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9 f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9 + exit 0

Notice the additional docker run of the weaveexec image half-way down, with --local dns-args and all the arguments we supplied. The result we get back is --dns 172.17.42.1 --dns-search=., which we insert into the docker run invocation of the application container further down.

So a single weave run command can result in running three containers: a short-lived weaveexec container to determine the DNS arguments, the actual application container, and a short-lived weaveexec container that attaches the application container to the Weave network.