Running a Weave Network on CoreOS
Introduction In this guide I will demonstrate how to provision Weave across a few CoreOS instances. In the last section I will show fully working example of automated provisioning by means of Cloud Config, which would allow you to get...
Introduction
In this guide I will demonstrate how to provision Weave across a few CoreOS instances. In the last section I will show fully working example of automated provisioning by means of Cloud Config, which would allow you to get started and move on to a larger deployment without having to figure out the basics. I assume you have already read some of the previous posts and have a good idea of how and where Weave is useful, and most certainly have good familiarity with CoreOS.
Step by Step
Firstly, we will need to implement a few systemd units that first bootstrap Weave network, and then start a set of test containers.
1. Download Weave Script and Container Image
There are probably alternative ways to get the script on the machine; e.g., if you have more scripts to install and have chosen to modify the weave script, you can probably clone a git repository with all of your scripts or sync an S3 bucket or do anything else. Below is one way to do it by means of Cloud Config and a systemd oneshot
unit that the services will depend on. I set RemainAfterExit
so this doesn’t get called more than once if it has few direct dependents.
This unit is needed on all nodes.
## /etc/systemd/system/install-weave.service
[Unit]
After=network-online.target
After=docker.service
Description=Install Weave
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin
https://raw.github.com/zettio/weave/master/weave
ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
ExecStartPre=/usr/bin/docker pull zettio/weave:latest
ExecStart=/bin/echo Wave Installed
2. Pre-download BusyBox Image
We will be starting a few containers as Weave nodes, which will use a BusyBox base image, so let’s pre-fetch it from the registry.
This unit is needed on all nodes.
## /etc/systemd/system/install-busybox.service
[Unit]
After=network-online.target
After=docker.service
Description=Install BusyBox
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker pull busybox:latest
3. Configure weave
Bridge Interface
NB: This is one important detail that took some time to figure out.
By default, CoreOS configures networkd to send DHCP queries on all interfaces and interfaces such as docker0
are white-listed; therefore the weave
interface needs to be white-listed too. It is anything but useful to have DHCP packets flowing through Weave network.
This unit is needed on all nodes.
## /etc/systemd/network/10-weave.network
[Match]
Type=bridge
Name=weave*
[Network]
The behaviour described above is mainly defined by the following configuration files shipped with CoreOS (current version 472.0.0):
4. Launch Weave Router Service Container
This unit goes on the first host in the cluster (host0
).
We first run weave launch
, which creates a container in the background, then attach it and run in foreground keeping the service persistent and logging to the system journal.
## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service
[Service]
ExecStartPre=/opt/bin/weave launch 10.0.0.1/16
ExecStart=/usr/bin/docker attach weave
To launch Weave routers on other hosts in the cluster you will need to know the IP address of host0
. This is how you would define service container on host1
, given that host0
’s IP is 192.168.0.101
:
## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service
[Service]
ExecStartPre=/opt/bin/weave launch 10.0.0.2/16 192.168.0.101
ExecStart=/usr/bin/docker attach weave
However, this is a little tedious and is striving for some code re-usability. With systemd one can make use of EnvironmentFile
directive, which plays rather nicely with hostname substitution (%H
). The example below shows how one can achieve this and also improves the configuration a little further by providing ExecStop
directive.
## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service
[Service]
EnvironmentFile=/etc/weave.%H.env
ExecStartPre=/opt/bin/weave launch $WEAVE_LAUNCH_ARGS
ExecStart=/usr/bin/docker logs -f weave
SuccessExitStatus=2
ExecStop=/opt/bin/weave stop
For the above to function, we need provide environment files for each of the hosts.
## /etc/weave.host0.env
WEAVE_LAUNCH_ARGS=""
## /etc/weave.host1.env
WEAVE_LAUNCH_ARGS="host0"
5. Launch Some Test Containers
Given a two-host cluster, this how we can start two containers that ping each other:
## /etc/systemd/system/pinger.service
[Unit]
After=weave.service
After=install-busybox.service
Description=Weave Network Test Monitor
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service
[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave
run $PINGER_LOCAL
--name=pinger busybox:latest
ping $PINGER_REMOTE
However while the above should demonstrate that our network if functional, it doesn’t represent anything close to a real-world application you are more likely to run. So here is an example of service that better represents what you might wish to run on your Weave network.
## /etc/systemd/system/greater.service
[Unit]
After=weave.service
After=install-busybox.service
Description=Weave Network Test Service
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service
[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave
run $GREATER_ADDRESS
--name=greater busybox:latest
nc -ll -p 2000 0.0.0.0 -e /bin/echo $GREATER_MESSAGE
In both example above we simply use oneshot
units, but for a real application you will probably want to configure it in the vein of the weave.service
unit.
For all of the above to function, we also need to amend the environment files.
## /etc/weave.host0.env
WEAVE_LAUNCH_ARGS=""
PINGER_LOCAL="10.0.1.1/24"
PINGER_REMOTE="10.0.1.2"
GREATER_ADDRESS="10.0.2.1/24"
GREATER_MESSAGE="Hello from #1"
## /etc/weave.host1.env
WEAVE_LAUNCH_ARGS="172.17.8.101"
PINGER_LOCAL="10.0.1.2/24"
PINGER_REMOTE="10.0.1.1"
GREATER_ADDRESS="10.0.2.2/24"
GREATER_MESSAGE="Hello from #2"
Wrap It All Together to Automate
You certainly don’t want to provision everything manually and the best way to do it with CoreOS is by using the Cloud Config provisioning mechanism that we have mentioned earlier. Here is what it might look like for you. Although, for your own application containers you might rather use Fleet, for the purpose of this guide, I thought I’d rather avoid adding Fleet into the mix.
#cloud-config
write_files:
- path: /etc/weave.core-01.env
permissions: 0644
owner: root
content: |
WEAVE_LAUNCH_ARGS=""
PINGER_LOCAL="10.0.1.1/24"
PINGER_REMOTE="10.0.1.2"
GREATER_ADDRESS="10.0.2.1/24"
GREATER_MESSAGE="Hello from #1"
- path: /etc/weave.core-02.env
permissions: 0644
owner: root
content: |
WEAVE_LAUNCH_ARGS="172.17.8.101"
PINGER_LOCAL="10.0.1.2/24"
PINGER_REMOTE="10.0.1.1"
GREATER_ADDRESS="10.0.2.2/24"
GREATER_MESSAGE="Hello from #2"
coreos:
units:
- name: 10-weave.network
runtime: false
content: |
[Match]
Type=bridge
Name=weave*
[Network]
- name: install-weave.service
command: start
enable: true
content: |
[Unit]
After=network-online.target
After=docker.service
Description=Install Weave
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin
https://raw.github.com/zettio/weave/master/weave
ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
ExecStartPre=/usr/bin/docker pull zettio/weave:latest
ExecStart=/bin/echo Wave Installed
- name: weave.service
command: start
enable: true
content: |
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service
[Service]
EnvironmentFile=/etc/weave.%H.env
ExecStartPre=/opt/bin/weave launch $WEAVE_LAUNCH_ARGS
ExecStart=/usr/bin/docker logs -f weave
SuccessExitStatus=2
ExecStop=/opt/bin/weave stop
- name: pinger.service
command: start
enable: true
content: |
[Unit]
After=weave.service
After=install-busybox.service
Description=Weave Network Test Monitor
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service
[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave
run $PINGER_LOCAL
--name=pinger busybox:latest
ping $PINGER_REMOTE
- name: greater.service
command: start
enable: true
content: |
[Unit]
After=weave.service
After=install-busybox.service
Description=Weave Network Test Service
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service
[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave
run $GREATER_ADDRESS
--name=greater busybox:latest
nc -ll -p 2000 0.0.0.0 -e /bin/echo $GREATER_MESSAGE
- name: install-busybox.service
command: start
enable: true
content: |
[Unit]
After=network-online.target
After=docker.service
Description=Install BusyBox
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker pull busybox:latest
This can be tested out of the box with Vagrant. I have used VirtualBox, but it should be easy to make it work with other Vagrant providers.
Without using Vagrant, you will need to either assign a static IP (or configure DNS) for core-01
and substitute that in WEAVE_LAUNCH_ARGS
for core-02
(instead of 172.17.8.101
, on line 17 in the example), so Weave router on core-02
can connect to core-01
.
You first will need to obtain coreos/coreos-vagrant
and then put the example above into user-data
file, then fire up the box.
git clone https://github.com/coreos/coreos-vagrant
cd coreos-vagrant
curl -o user-data https://gist.githubusercontent.com/errordeveloper/46e20edc39d7e9fe2e82/raw/config.yaml
echo '$num_instances=2' > config.rb
vagrant up
You should now have two boxes – core-01
and core-02
.
Now you can log on to each of the VMs and verify the setup is complete. First you want to run docker ps
, which should show 3 containers running — if you see fewer or none, repeat the command until they appear. If the containers do not appear, something may have gone wrong, so you want to check systemctl status weave
, and maybe check for any error in the logs using journalctl
command.
> vagrant ssh core-01
CoreOS (alpha)
core@core-01 ~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3dba5a8c6806 busybox:latest "nc -ll -p 2000 0.0. 10 seconds ago Up 10 seconds greater
15790c3d3b2b busybox:latest "ping 10.0.1.2" 11 seconds ago Up 10 seconds pinger
9ad4c9bfbd24 zettio/weave:git-a1c4038ac765 "/home/weave/weaver 15 seconds ago Up 14 seconds 0.0.0.0:6783->6783/udp, 0.0.0.0:6783->6783/tcp weave
core@core-01 ~ $ docker logs pinger | tail -n 5
64 bytes from 10.0.1.2: seq=28 ttl=64 time=1.116 ms
64 bytes from 10.0.1.2: seq=29 ttl=64 time=1.177 ms
64 bytes from 10.0.1.2: seq=30 ttl=64 time=1.109 ms
64 bytes from 10.0.1.2: seq=31 ttl=64 time=1.115 ms
64 bytes from 10.0.1.2: seq=32 ttl=64 time=1.095 ms
core@core-01 ~ $
You can repeat this on the core-02
VM as well now.
Once you verified that the ping container on core-01
can reach its friend on core-02
via the Weave network, you can start an interactive test container on either of the hosts.
core@core-01 ~ $ sudo weave run 10.0.2.101/24 -ti --name=client1 busybox:latest /bin/sh
a8160938472fff75241246f684657ec2efe1e2554eb666d140bd05565bf7dc4b
core@core-01 ~ $ docker attach client1
/ # nc 10.0.2.1 2000
Hello from #1
/ # nc 10.0.2.2 2000
Hello from #2
/ # exit
core@core-01 ~ $
Conclusion
You should now be able to take this to the next step and deploy your own services on Weave network using CoreOS!
Follow-up
Luke Bond have posted a cleaner Cloud Config template without the extra examples, on which you can base your own setup.