Running Containers on Remote Docker Hosts with Weave 0.10
In an earlier post we peeked under the covers of Weave 0.10 to discover how it interacts with remote Docker hosts. So, for example weave launch is translated into: <br> docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v...
In an earlier post we peeked under the covers of Weave 0.10 to discover how it interacts with remote Docker hosts. So, for example weave launch
is translated into:
<br>
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local launch<br>
Other weave commands are translated in the same way. Except weave run
. Following the above pattern we would expect weave run 10.2.0.1/24 -ti ubuntu
to turn into:
<br>
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local run 10.2.0.1/24 -ti ubuntu<br>
But that is not what happens…
Stuck between versions
The above translation of weave run
does in fact work, but only if all the run
options specified are supported by Docker 1.3.1. That is because the weaveexec
image embeds a Docker client of that version. We chose such a relatively old version because we want Weave to run in places that may not have the latest Docker version installed. Docker daemons are backward compatible, so an old client can talk to a newer daemon. Hence our choice of Docker version in the weaveexec
image means Weave will work with all Docker daemons >= 1.3.1. But the downside is that any interactions with Docker through weaveexec
must use options that the embedded Docker 1.3.1 understands.
So how do we let users employ features of Docker versions > 1.3.1? This is most pertinent to weave run
, which is the primary means of starting application containers attached to the Weave network. The options to docker run
, which it invokes, have evolved significantly between Docker versions. So, for example, a user running Docker 1.6 may want to attach a container label – a new feature in 1.6 – with --label=...
when invoking weave run
. If weave run
was executing via the weaveexec
image as shown above, that would result in an error, since the Docker 1.3.1 client embedded in the image wouldn’t know how to parse the --label=...
parameter.
In a way we are trying to satisify an impossible constraint: we want the weaveexec
Docker version to be old, so it works with old daemons, and new, so it supports recently introduced docker run
options.
2 for 1
The solution we implemented in the weave script is to translate weave run
into a combination of docker run
and weave attach
. The former bypasses the weaveexec
image, and hence is not constrained by that image’s Docker version.
Let’s see what an execution of weave run 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu
actually looks like…
<br>
$ sh -x weave run 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ set -e<br>
+ SCRIPT_VERSION=0.10.0<br>
+ [ 0.10.0 = (unreleased version) ]<br>
+ IMAGE_VERSION=0.10.0<br>
+ IMAGE_VERSION=0.10.0<br>
+ DOCKERHUB_USER=weaveworks<br>
+ BASE_EXEC_IMAGE=weaveworks/weaveexec<br>
+ EXEC_IMAGE=weaveworks/weaveexec:0.10.0<br>
+ PROCFS=/proc<br>
+ [ run = run ]<br>
+ shift 1<br>
+ check_docker_version<br>
+ docker -v<br>
+ sed -n -e s|^Docker version ([0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*).*|1|p<br>
+ DOCKER_VERSION=1.6.0<br>
+ [ -z 1.6.0 ]<br>
+ [ 1.6.0 = 1.3.0 ]<br>
+ [ 10.2.0.1/24 = --with-dns ]<br>
+ collect_cidr_args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ CIDR_ARGS=<br>
+ CIDR_COUNT=0<br>
+ is_cidr 10.2.0.1/24<br>
+ echo 10.2.0.1/24<br>
+ grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$<br>
+ CIDR_ARGS= 10.2.0.1/24<br>
+ CIDR_COUNT=1<br>
+ shift 1<br>
+ is_cidr --label=com.foo.deployment=stage<br>
+ echo --label=com.foo.deployment=stage<br>
+ grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$<br>
+ [ 1 -eq 0 ]<br>
+ shift 1<br>
+ docker run -d --label=com.foo.deployment=stage -ti ubuntu<br>
+ CONTAINER=0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa<br>
+ exec_remote attach 10.2.0.1/24 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa<br>
+ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local attach 10.2.0.1/24 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa<br>
+ echo 0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa<br>
0eedd4c8c0a53d53209f415f5d1f004d75163f94a3f906d652aa7b37e56bbaaa<br>
+ exit 0<br>
There is a lot more going on here than in weave launch
trace we saw earlier, though it is not nearly as complicated as it may first appear. The initial execution path, up to the setting of EXEC_IMAGE
is the same as for weave launch
. Next the script checks that we are not running Docker 1.3.1, which contains a nasty bug that prevents weave run
from working. Then the script strips off all the weave specific parameters, such as the --with-dns
flag and CIDR args (in this case, the 10.2.0.1/24
we supplied). It then invokes docker run -d
to start the application container in the background, with all the non-weave arguments. The resulting container id is given to weave attach
, together with the collected CIDRs. The weave attach
is translated into an invocation of a container running the weaveexec
image, exactly as we have seen previously. Finally, we echo the container id back to the user.
All pretty simple, really, and fairly easy to grasp from the code in the relevant section of the script:
<br>
if [ "$1" = "run" ]; then<br>
shift 1<br>
check_docker_version<br>
if [ "$1" = "--with-dns" ]; then<br>
shift 1<br>
DNS_ARGS=$(exec_remote dns-args "$@")<br>
fi<br>
collect_cidr_args "$@"<br>
shift $CIDR_COUNT<br>
CONTAINER=$(docker run $DNS_ARGS -d "$@")<br>
exec_remote attach $CIDR_ARGS $CONTAINER<br>
echo $CONTAINER<br>
exit 0<br>
...<br>
But the astute reader will notice another complication in the above…
3 containers for 1 command
weave run
with the --with-dns
flag requires starting the application container with extra dns-related parameters to docker run
. Figuring our what those should be is not completely trivial, and requires parsing of the supplied arguments, and determining the IP of the docker bridge. The script is delegating that task to the weaveexec
container, invoking the equivalent of weave dns-args ...
. The output contains the dns-related parameters, which are spliced into the invocation of docker run
.
Here’s what a trace of weave run --with-dns 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu
looks like…
<br>
$ sh -x weave run --with-dns 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ set -e<br>
+ SCRIPT_VERSION=0.10.0<br>
+ [ 0.10.0 = (unreleased version) ]<br>
+ IMAGE_VERSION=0.10.0<br>
+ IMAGE_VERSION=0.10.0<br>
+ DOCKERHUB_USER=weaveworks<br>
+ BASE_EXEC_IMAGE=weaveworks/weaveexec<br>
+ EXEC_IMAGE=weaveworks/weaveexec:0.10.0<br>
+ PROCFS=/proc<br>
+ [ run = run ]<br>
+ shift 1<br>
+ check_docker_version<br>
+ docker -v<br>
+ sed -n -e s|^Docker version ([0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*).*|1|p<br>
+ DOCKER_VERSION=1.6.0<br>
+ [ -z 1.6.0 ]<br>
+ [ 1.6.0 = 1.3.0 ]<br>
+ [ --with-dns = --with-dns ]<br>
+ shift 1<br>
+ exec_remote dns-args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local dns-args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ DNS_ARGS=--dns 172.17.42.1 --dns-search=.<br>
+ collect_cidr_args 10.2.0.1/24 --label=com.foo.deployment=stage -ti ubuntu<br>
+ CIDR_ARGS=<br>
+ CIDR_COUNT=0<br>
+ is_cidr 10.2.0.1/24<br>
+ echo 10.2.0.1/24<br>
+ grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$<br>
+ CIDR_ARGS= 10.2.0.1/24<br>
+ CIDR_COUNT=1<br>
+ shift 1<br>
+ is_cidr --label=com.foo.deployment=stage<br>
+ echo --label=com.foo.deployment=stage<br>
+ grep -E ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}$<br>
+ [ 1 -eq 0 ]<br>
+ shift 1<br>
+ docker run --dns 172.17.42.1 --dns-search=. -d --label=com.foo.deployment=stage -ti ubuntu<br>
+ CONTAINER=f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9<br>
+ exec_remote attach 10.2.0.1/24 f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9<br>
+ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e VERSION -e WEAVE_DOCKER_ARGS -e WEAVEDNS_DOCKER_ARGS -e WEAVE_PASSWORD -e WEAVE_PORT -e WEAVE_CONTAINER_NAME -e DOCKER_BRIDGE -e PROCFS=/hostproc --privileged --net=host weaveworks/weaveexec:0.10.0 --local attach 10.2.0.1/24 f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9<br>
+ echo f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9<br>
f4e6e596d10c383466966bfadc7f01c913c275dab148cd3153d4d44830c7a2a9<br>
+ exit 0<br>
Notice the additional docker run
of the weaveexec image half-way down, with --local dns-args
and all the arguments we supplied. The result we get back is --dns 172.17.42.1 --dns-search=.
, which we insert into the docker run
invocation of the application container further down.
So a single weave run
command can result in running three containers: a short-lived weaveexec
container to determine the DNS arguments, the actual application container, and a short-lived weaveexec
container that attaches the application container to the Weave network.