From the Blog:

Running a Weave Network on CoreOS

October 28, 2014
guide

Introduction In this guide I will demonstrate how to provision Weave across a few CoreOS instances. In the last section I will show fully working example of automated provisioning by means of Cloud Config, which would allow you to get...

Related posts

Weave Discovery and Docker Swarm

Read more

Docker networking 1.9 and Weave technical deep-dive

Read more

Service Discovery and Load Balancing with Weave on Amazon's ECS

Read more

Introduction

In this guide I will demonstrate how to provision Weave across a few CoreOS instances. In the last section I will show fully working example of automated provisioning by means of Cloud Config, which would allow you to get started and move on to a larger deployment without having to figure out the basics. I assume you have already read some of the previous posts and have a good idea of how and where Weave is useful, and most certainly have good familiarity with CoreOS.

Step by Step

Firstly, we will need to implement a few systemd units that first bootstrap Weave network, and then start a set of test containers.

1. Download Weave Script and Container Image

There are probably alternative ways to get the script on the machine; e.g., if you have more scripts to install and have chosen to modify the weave script, you can probably clone a git repository with all of your scripts or sync an S3 bucket or do anything else. Below is one way to do it by means of Cloud Config and a systemd oneshot unit that the services will depend on. I set RemainAfterExit so this doesn’t get called more than once if it has few direct dependents.

This unit is needed on all nodes.

## /etc/systemd/system/install-weave.service
[Unit]
After=network-online.target
After=docker.service
Description=Install Weave
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/bin/wget -N -P /opt/bin 
    https://raw.github.com/zettio/weave/master/weave 
ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
ExecStartPre=/usr/bin/docker pull zettio/weave:latest
ExecStart=/bin/echo Wave Installed

2. Pre-download BusyBox Image

We will be starting a few containers as Weave nodes, which will use a BusyBox base image, so let’s pre-fetch it from the registry.

This unit is needed on all nodes.

## /etc/systemd/system/install-busybox.service
[Unit]
After=network-online.target
After=docker.service
Description=Install BusyBox
Documentation=http://zettio.github.io/weave/
Requires=network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker pull busybox:latest

3. Configure weave Bridge Interface

NB: This is one important detail that took some time to figure out.

By default, CoreOS configures networkd to send DHCP queries on all interfaces and interfaces such as docker0 are white-listed; therefore the weave interface needs to be white-listed too. It is anything but useful to have DHCP packets flowing through Weave network.

This unit is needed on all nodes.

## /etc/systemd/network/10-weave.network
[Match]
Type=bridge
Name=weave*

[Network]

The behaviour described above is mainly defined by the following configuration files shipped with CoreOS (current version 472.0.0):

4. Launch Weave Router Service Container

This unit goes on the first host in the cluster (host0).

We first run weave launch, which creates a container in the background, then attach it and run in foreground keeping the service persistent and logging to the system journal.

## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service

[Service]
ExecStartPre=/opt/bin/weave launch 10.0.0.1/16
ExecStart=/usr/bin/docker attach weave

To launch Weave routers on other hosts in the cluster you will need to know the IP address of host0. This is how you would define service container on host1, given that host0’s IP is 192.168.0.101:

## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service

[Service]
ExecStartPre=/opt/bin/weave launch 10.0.0.2/16 192.168.0.101
ExecStart=/usr/bin/docker attach weave

However, this is a little tedious and is striving for some code re-usability. With systemd one can make use of EnvironmentFile directive, which plays rather nicely with hostname substitution (%H). The example below shows how one can achieve this and also improves the configuration a little further by providing ExecStop directive.

## /etc/systemd/system/weave.service
[Unit]
After=install-weave.service
Description=Weave Network
Documentation=http://zettio.github.io/weave/
Requires=install-weave.service

[Service]
EnvironmentFile=/etc/weave.%H.env
ExecStartPre=/opt/bin/weave launch $WEAVE_LAUNCH_ARGS
ExecStart=/usr/bin/docker logs -f weave
SuccessExitStatus=2
ExecStop=/opt/bin/weave stop

For the above to function, we need provide environment files for each of the hosts.

## /etc/weave.host0.env
WEAVE_LAUNCH_ARGS=""

## /etc/weave.host1.env
WEAVE_LAUNCH_ARGS="host0"

5. Launch Some Test Containers

Given a two-host cluster, this how we can start two containers that ping each other:

## /etc/systemd/system/pinger.service
[Unit]
After=weave.service 
After=install-busybox.service
Description=Weave Network Test Monitor
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service

[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave 
  run $PINGER_LOCAL 
  --name=pinger busybox:latest 
  ping $PINGER_REMOTE

However while the above should demonstrate that our network if functional, it doesn’t represent anything close to a real-world application you are more likely to run. So here is an example of service that better represents what you might wish to run on your Weave network.

## /etc/systemd/system/greater.service
[Unit]
After=weave.service 
After=install-busybox.service
Description=Weave Network Test Service
Documentation=http://zettio.github.io/weave/
Requires=weave.service
Requires=install-busybox.service

[Service]
EnvironmentFile=/etc/weave.%H.env
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/weave 
  run $GREATER_ADDRESS 
  --name=greater busybox:latest 
  nc -ll -p 2000 0.0.0.0 -e /bin/echo $GREATER_MESSAGE

In both example above we simply use oneshot units, but for a real application you will probably want to configure it in the vein of the weave.service unit.

For all of the above to function, we also need to amend the environment files.

## /etc/weave.host0.env
WEAVE_LAUNCH_ARGS=""
PINGER_LOCAL="10.0.1.1/24"
PINGER_REMOTE="10.0.1.2"
GREATER_ADDRESS="10.0.2.1/24"
GREATER_MESSAGE="Hello from #1"

## /etc/weave.host1.env
WEAVE_LAUNCH_ARGS="172.17.8.101"
PINGER_LOCAL="10.0.1.2/24"
PINGER_REMOTE="10.0.1.1"
GREATER_ADDRESS="10.0.2.2/24"
GREATER_MESSAGE="Hello from #2"

Wrap It All Together to Automate

You certainly don’t want to provision everything manually and the best way to do it with CoreOS is by using the Cloud Config provisioning mechanism that we have mentioned earlier. Here is what it might look like for you. Although, for your own application containers you might rather use Fleet, for the purpose of this guide, I thought I’d rather avoid adding Fleet into the mix.

#cloud-config
write_files:
  - path: /etc/weave.core-01.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_LAUNCH_ARGS=""
      PINGER_LOCAL="10.0.1.1/24"
      PINGER_REMOTE="10.0.1.2"
      GREATER_ADDRESS="10.0.2.1/24"
      GREATER_MESSAGE="Hello from #1"

  - path: /etc/weave.core-02.env
    permissions: 0644
    owner: root
    content: |
      WEAVE_LAUNCH_ARGS="172.17.8.101"
      PINGER_LOCAL="10.0.1.2/24"
      PINGER_REMOTE="10.0.1.1"
      GREATER_ADDRESS="10.0.2.2/24"
      GREATER_MESSAGE="Hello from #2"


coreos:
  units:
    - name: 10-weave.network
      runtime: false
      content: |
        [Match]
        Type=bridge
        Name=weave*

        [Network]

    - name: install-weave.service
      command: start
      enable: true
      content: |
        [Unit]
        After=network-online.target
        After=docker.service
        Description=Install Weave
        Documentation=http://zettio.github.io/weave/
        Requires=network-online.target
        Requires=docker.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStartPre=/usr/bin/wget -N -P /opt/bin 
            https://raw.github.com/zettio/weave/master/weave 
        ExecStartPre=/usr/bin/chmod +x /opt/bin/weave
        ExecStartPre=/usr/bin/docker pull zettio/weave:latest
        ExecStart=/bin/echo Wave Installed

    - name: weave.service
      command: start
      enable: true
      content: |
        [Unit]
        After=install-weave.service
        Description=Weave Network
        Documentation=http://zettio.github.io/weave/
        Requires=install-weave.service

        [Service]
        EnvironmentFile=/etc/weave.%H.env
        ExecStartPre=/opt/bin/weave launch $WEAVE_LAUNCH_ARGS
        ExecStart=/usr/bin/docker logs -f weave
        SuccessExitStatus=2
        ExecStop=/opt/bin/weave stop

    - name: pinger.service
      command: start
      enable: true
      content: |
        [Unit]
        After=weave.service 
        After=install-busybox.service
        Description=Weave Network Test Monitor
        Documentation=http://zettio.github.io/weave/
        Requires=weave.service
        Requires=install-busybox.service

        [Service]
        EnvironmentFile=/etc/weave.%H.env
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/opt/bin/weave 
          run $PINGER_LOCAL 
          --name=pinger busybox:latest 
          ping $PINGER_REMOTE

    - name: greater.service
      command: start
      enable: true
      content: |
        [Unit]
        After=weave.service 
        After=install-busybox.service
        Description=Weave Network Test Service
        Documentation=http://zettio.github.io/weave/
        Requires=weave.service
        Requires=install-busybox.service

        [Service]
        EnvironmentFile=/etc/weave.%H.env
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/opt/bin/weave 
          run $GREATER_ADDRESS 
          --name=greater busybox:latest 
          nc -ll -p 2000 0.0.0.0 -e /bin/echo $GREATER_MESSAGE

    - name: install-busybox.service
      command: start
      enable: true
      content: |
        [Unit]
        After=network-online.target
        After=docker.service
        Description=Install BusyBox
        Documentation=http://zettio.github.io/weave/
        Requires=network-online.target
        Requires=docker.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/docker pull busybox:latest

This can be tested out of the box with Vagrant. I have used VirtualBox, but it should be easy to make it work with other Vagrant providers.

Without using Vagrant, you will need to either assign a static IP (or configure DNS) for core-01 and substitute that in WEAVE_LAUNCH_ARGS for core-02 (instead of 172.17.8.101, on line 17 in the example), so Weave router on core-02 can connect to core-01.

You first will need to obtain coreos/coreos-vagrant and then put the example above into user-data file, then fire up the box.

git clone https://github.com/coreos/coreos-vagrant
cd coreos-vagrant
curl -o user-data https://gist.githubusercontent.com/errordeveloper/46e20edc39d7e9fe2e82/raw/config.yaml
echo '$num_instances=2' > config.rb
vagrant up

You should now have two boxes – core-01 and core-02.

Now you can log on to each of the VMs and verify the setup is complete. First you want to run docker ps, which should show 3 containers running — if you see fewer or none, repeat the command until they appear. If the containers do not appear, something may have gone wrong, so you want to check systemctl status weave, and maybe check for any error in the logs using journalctl command.

> vagrant ssh core-01
CoreOS (alpha)
core@core-01 ~ $ docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS              PORTS                                            NAMES
3dba5a8c6806        busybox:latest                  "nc -ll -p 2000 0.0.   10 seconds ago      Up 10 seconds                                                        greater             
15790c3d3b2b        busybox:latest                  "ping 10.0.1.2"        11 seconds ago      Up 10 seconds                                                        pinger              
9ad4c9bfbd24        zettio/weave:git-a1c4038ac765   "/home/weave/weaver    15 seconds ago      Up 14 seconds       0.0.0.0:6783->6783/udp, 0.0.0.0:6783->6783/tcp   weave               
core@core-01 ~ $ docker logs pinger | tail -n 5
64 bytes from 10.0.1.2: seq=28 ttl=64 time=1.116 ms
64 bytes from 10.0.1.2: seq=29 ttl=64 time=1.177 ms
64 bytes from 10.0.1.2: seq=30 ttl=64 time=1.109 ms
64 bytes from 10.0.1.2: seq=31 ttl=64 time=1.115 ms
64 bytes from 10.0.1.2: seq=32 ttl=64 time=1.095 ms
core@core-01 ~ $

You can repeat this on the core-02 VM as well now.

Once you verified that the ping container on core-01 can reach its friend on core-02 via the Weave network, you can start an interactive test container on either of the hosts.

core@core-01 ~ $ sudo weave run 10.0.2.101/24 -ti --name=client1 busybox:latest /bin/sh
a8160938472fff75241246f684657ec2efe1e2554eb666d140bd05565bf7dc4b
core@core-01 ~ $ docker attach client1

/ # nc 10.0.2.1 2000
Hello from #1
/ # nc 10.0.2.2 2000
Hello from #2
/ # exit
core@core-01 ~ $

Conclusion

You should now be able to take this to the next step and deploy your own services on Weave network using CoreOS!

Follow-up

Luke Bond have posted a cleaner Cloud Config template without the extra examples, on which you can base your own setup.

 

Related posts

Weave Discovery and Docker Swarm

Read more

Docker networking 1.9 and Weave technical deep-dive

Read more

Service Discovery and Load Balancing with Weave on Amazon's ECS

Read more

Previous
usecase

Walkthrough of Connecting Nginx, PHP and MySQL with Weave
Next
implementation

Why Do We Have a Script that Wraps the 'docker' Command?
Previous
usecase

Walkthrough of Connecting Nginx, PHP and MySQL with Weave
Next
implementation

Why Do We Have a Script that Wraps the 'docker' Command?