Microservices deployment on Amazon EC2 Container Service with Weave Net

Amazon EC2 container service is a scalable container management service that allows you to manage Docker containers on a cluster of Amazon EC2 instances. Weave Net provides a simple and robust software-defined network for apps running as Docker containers that does not require a external database (cluster store). Weave simplifies setting up a container network within the Amazon EC2 Container Service. Because Weave allows each container to use standard port numbers – for example you can expose MySQL on port 3306 – managing services is straightforward. Every container can find the IP of any other container using a simple DNS query on the container’s name, and communicate directly without NAT or complicated port mapping.

An advantage to using DNS is that when you use a container name within (say) a config file, you are not required to have a script in place to generate the name based on run-time variables. You can also optionally burn the config file with the hostname right into the container image.

There is no need to deploy extra services to achieve DNS lookup and load balancing. Weave Net takes care of both automatic service discovery and load balancing, reducing overhead and complexity. Weave saves you time and money, and lets you focus on app development rather than your infrastructure design.

About This Example

This guide takes approximately 15 minutes to complete: you will use Weave for service discovery and load balancing between containers that have been deployed to Amazon Elastic Cloud (EC2) instances using Amazon Container Service or ECS.

This guide also introduces Weave Scope, and Weave Cloud which enables you to visualize and understand your container-based microservices.

Two types of containerized microservices are demonstrated in this guide: HTTP Servers and “Data Producers”.

AWS ECS overview diagram

Data producers generically model containers that produce a data feed of some kind. The HTTP Servers present a web interface to the data from the Data Producers. This is a very common pattern in distributed systems, but its implementation requires answers to the following questions:

Service discovery: How does an HTTP Server find a Data Producer to connect to?
Load balancing/fault tolerance: How can the HTTP Servers make uniform and efficient use of all the Data Producers available to them?

Weave solves these issues using its built-in DNS server, where it securely and transparently:

Implements service discovery by adding DNS A-records for your containers based on their names.
Manages load balancing by randomizing the order of DNS responses.

For more information about the weaveDNS service see Automatic Discovery with weaveDNS

What You Will Use

Before You Begin

We will make use of the Amazon Web Services (AWS) CLI tool to manage and access AWS. You need a valid Amazon Web Services account, and also have the AWS CLI set up and configured before working through this guide. Please ensure your AWS account has the appropriate administrative privileges. Amazon provides extensive documentation on how to set up the AWS CLI.

Getting the Code

The code for this guide is available on GitHub. Clone the weaveworks/guides repository and then change to the aws-ecs directory, from which you will be working throughout most of this guide.

git clone https://github.com/weaveworks/guides
cd guides/aws-ecs

AWS-CLI Configuration

Make sure AWS-CLI is set to use a region where Amazon ECS is available (us-east-1, us-west1, us-west-2, eu-west-1, ap-northeast-1 or ap-southeast-2 at the time of writing this guide). See Regions & Endpoints documentation at Amazon Web Service for more information.

View AWS-CLI’s configuration with

aws configure list

and then modify it by running:

aws configure

Obtain a Weave Cloud Service Token (Recommended)

To visualize this example with the Weave Cloud service, you will need to obtain a Weave Cloud service token.

To gain access, sign up at Weave Cloud. An email will be sent to you with further login instructions.

Weave Cloud

Automatic Setup and Configuration

To configure this example, run the following command:

./setup.sh $SCOPE_TOKEN

where,

$SCOPE_TOKEN is an optional argument corresponding to your Weave Cloud service token.

You should see something like this:

    Creating ECS cluster (weave-ecs-demo-cluster) .. done
    Creating VPC (weave-ecs-demo-vpc) .. done
    Creating Subnet (weave-ecs-demo-subnet) .. done
    Creating Internet Gateway (weave-ecs-demo) .. done
    Creating Security Group (weave-ecs-demo) .. done
    Creating Key Pair (weave-ecs-demo, file weave-ecs-demo-key.pem) .. done
    Creating IAM role (weave-ecs-role) .. done
    Creating Launch Configuration (weave-ecs-launch-configuration) .. done
    Creating Auto Scaling Group (weave-ecs-demo-group) with 3 instances .. done
    Waiting for instances to join the cluster (this may take a few minutes) .. done
    Registering ECS Task Definition (weave-ecs-demo-task) .. done
    Creating ECS Service with 3 tasks (weave-ecs-demo-service) .. done
    Waiting for tasks to start running .. done
    Setup is ready!
    Open your browser and go to any of these URLs:
      http://foo.region.compute.amazonaws.com
      http://bar.region.compute.amazonaws.com
      http://baz.region.compute.amazonaws.com

What Just Happened?

The setup.sh script automatically:

Created an Amazon ECS cluster named weave-ecs-demo-cluster
Spawned three hosts (EC2 instances), which are now part of the cluster and are based on Weave’s ECS AMI.
Created an ECS task family definition that describes the HTTP Server and “Data Producer” containers.
Spawned three tasks, one per host, resulting in an HTTP Server and a Data Producer container running on each host.

Testing the Setup

The three URLs shown above communicate via your browser with the HTTP Server containers. Pick one of them (or all three, if you like) and open them in your browser:

  http://foo.region.compute.amazonaws.com
  http://bar.region.compute.amazonaws.com
  http://baz.region.compute.amazonaws.com

This is what you should see:

Reload your browser to force the HTTP Server to refresh its Data Provider address list, balancing the load between the EC2 instances.

How Do Service Discovery and Load Balancing Work?

Both the HTTP Server and the Data Producer containers are very simple. They were implemented with a few lines of bash, using Netcat.

Container dataproducer:

while true; do
  IP=`hostname -i | awk '{ print $1 }'`
  echo "Hi, this is the data producer in $IP" | nc -q 0 -l -p 4540
done

Container httpserver:

sleep 7 # Wait for data producers to start
  while true; do
  # Get a message from a data producer
  DATA_PRODUCER_MESSAGE=`nc dataproducer 4540`
  HTML="<html> <head> <title>Weaveworks Amazon ECS Sample App<\/title> <style>body {margin-top:   40px; background-color: #333;} <\/style> <\/head><body> <div style=color:white;text-align:center> <h1>Chosen data producer message:<\/h1> <h2>${DATA_PRODUCER_MESSAGE}<\/h2> <\/div>"
  echo "$HTML" | nc -q 0 -l -p 80
done

Note: the source code shown above has been simplified and reformatted for clarity.

ECS and Weave Diagram

When ECS launches a container, the call to Docker is intercepted by Weave’s Docker API proxy, and an address is assigned using Weave’s automatic IP allocator. The container is then registered with the Weave DNS service and attached to the Weave network. Weave Net’s DNS registers A-records based on the container’s name:

A dataproducer A-record for all the Data Producer containers.
A httpserver A-record for all the HTTP Server container.

The Data Producer waits for requests on TCP port 4540 and responds with a string containing its IP address.

The HTTP Server works as follows:

Contacts a Data Producer and obtains its message (nc dataproducer 4540). This implicitly does the load balancing due to weavedns’ response randomization (more about this in the next section).
Composes HTML with the information obtained in (1) and (2).
Waits for a browser to connect.

What’s Happening in the Hosts?

Weave Scope provides a real-time visualization of the containers running on the Weave network and also an overview of how they communicate with each other.

If you provided a $PROBE_TOKEN to setup.sh, simply login to https://cloud.weave.works
As an alternative, if you did not provide a token, all of your ECS instances will still be running equipped with a Weave Scope web application listening on port 4040 (this solution does have some performance and administration issues compared to using the Weave Cloud service, but it is more than sufficient for demonstration purposes).

To access it, open your browser and paste the Weave Scope application URL from any of your instances:

      http://foo.region.compute.amazonaws.com:4040
      http://bar.region.compute.amazonaws.com:4040
      http://baz.region.compute.amazonaws.com:4040

This is what you should see with Weave Scope when accessing one of the HTTP Servers multiple times (i.e. reloading http://foo.region.compute.amazonaws.com in your browser multiple times).

Weave Scope Visualization

Click the httpserver container to display its details.

Note that the edges between one of the httpserver containers (the one accessed from your browser) and the three dataproducer containers, reflect the load balancing scheme that we built using the weavedns service.

More insights into the Weave network can be gleaned by selecting different views: Applications (by name), Containers by image and Hosts.

To dig even deeper on what’s happening in the ECS instances, access them through ssh:

    ssh -i weave-ecs-demo-key.pem ec2-user@${INSTANCE}

where ${INSTANCE} can be any of your 3 instance hostnames (foo.region.compute.amazonaws.com, bar.region.compute.amazonaws.com or baz.region.compute.amazonaws.com in the setup.sh example listing above).

For example, to list the active running containers in the instance:

[ec2-user@ip-XXX-XXX-XXX-XXX ~]$ docker ps

Where you will see something similar to this:

    CONTAINER ID     IMAGE                            COMMAND                CREATED           STATUS           PORTS                          NAMES
    a67655146b5b     2opremio/weaveecsdemo:latest     "\"/w/w bash -c 'set   7 minutes ago     Up 7 minutes     0.0.0.0:80->80/tcp             ecs-weave-ecs-demo-task-1-httpserver-a2bad7f8f792f185f901
    eeb53274c26b     2opremio/weaveecsdemo:latest     "/w/w sh -c 'while t   7 minutes ago     Up 7 minutes                                    ecs-weave-ecs-demo-task-1-dataproducer-dec2b39a92e0edb1aa01
    8af86be1dd18     amazon/amazon-ecs-agent:latest   "/w/w /agent"          8 minutes ago     Up 8 minutes     127.0.0.1:51678->51678/tcp     ecs-agent
    693ef5ae00cb     weaveworks/weaveexec:v1.1.0      "/home/weave/weavepr   8 minutes ago     Up 8 minutes                                    weaveproxy
    86b6019c3995     weaveworks/weave:v1.1.0          "/home/weave/weaver    8 minutes ago     Up 8 minutes                                    weave
    190afc5cd56b     weaveworks/scope:latest          "/home/weave/entrypo   8 minutes ago     Up 8 minutes                                    weavescope

Container ecs-weave-ecs-demo-task-1-httpserver-a2bad7f8f792f185f901 is the HTTP Server of this host, producing the output you saw in your browser. Note how container names are mangled by ECS: ecs-${TASK_FAMILY_NAME}-${TASK_FAMILY_VERSION}-${STRIPPED_CONTAINER_NAME}-${UUID}.
Container ecs-weave-ecs-demo-task-1-dataproducer-dec2b39a92e0edb1aa01 is the Data Producer of this host.
Containers weaveproxy and weave are responsible for running Weave within each ECS instance. For illustration purposes, the proxy was shown out of Docker in the previous section’s diagram, but in actual fact weaveproxy runs inside of Docker.
Container weavescope is responsible for running Weave Scope and for monitoring each instance.
Container ecs-agent corresponds to Amazon’s ECS Agent, which runs on all of the EC2 instances and is responsible for starting containers on behalf of Amazon ECS. For simplification, the ECS Agent was represented out of Docker in the previous section’s diagram.

View the IP addresses of the HTTP Servers and the Data Producers by running:

    [ec2-user@ip-XXX-XXX-XXX-XXX ~]$ eval $(weave env) # Use weave-proxy
    [ec2-user@ip-XXX-XXX-XXX-XXX ~]$ docker run 2opremio/weaveecsdemo dig +short httpserver
    10.36.0.3
    10.32.0.3
    10.40.0.2
    [ec2-user@ip-XXX-XXX-XXX-XXX ~]$ docker run 2opremio/weaveecsdemo dig +short dataproducer
    10.36.0.2
    10.32.0.2
    10.40.0.1

Re-running these commands vary the IP addresses. This is the weavedns service transparently balancing the load by randomizing the IP addresses, as the HTTP servers are connecting to Data Producers.

Cleanup

To clean up this demonstration run:

./cleanup.sh

This script will work even if something has gone wrong while configuring the demonstration (for example, if setup.sh didn’t finish due to missing AWS permissions). If this was the case, cleanup.sh may output errors while it is trying to destroy resources, which weren’t created. Simply disregard them.

Manual Setup

To manually reproduce what ./setup.sh does automatically:

1. Create the ECS cluster

aws ecs create-cluster --cluster-name weave-ecs-demo-cluster

2. Create a Virtual Private Cloud

Create a VPC, which is necessary for creating t2.micro instances.

VPC_ID=$(aws ec2 create-vpc --cidr-block 172.31.0.0/28 --query 'Vpc.VpcId' --output text)

Enable DNS to get nice FQDNs for your instances.

aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-hostnames

Tag the VPC to enable automatic deletion with the cleanup.sh.

aws ec2 create-tags --resources $VPC_ID --tag Key=Name,Value=weave-ecs-demo-vpc

3. Create a subnet

Create a subnet to enable networking in your instances.

SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 172.31.0.0/28 --query 'Subnet.SubnetId' --output text)

Tag the subnet to enable automatic deletion with the cleanup.sh.

aws ec2 create-tags --resources $SUBNET_ID --tag Key=Name,Value=weave-ecs-demo-subnet

4. Create an Internet Gateway

Create an Internet Gateway so that you can access your EC2 instances externally.

GW_ID=$(aws ec2 create-internet-gateway --query 'InternetGateway.InternetGatewayId' --output text)

Attach the gateway to the VPC and ensure that the gateway is used as the destination of the default route.

aws ec2 attach-internet-gateway --internet-gateway-id $GW_ID --vpc-id $VPC_ID
TABLE_ID=$(aws ec2 describe-route-tables --query 'RouteTables[?VpcId==`'$VPC_ID'`].RouteTableId' --output text)
aws ec2 create-route --route-table-id $TABLE_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $GW_ID

Tag the gateway to enable automatic deletion with the cleanup.sh script.

aws ec2 create-tags --resources $GW_ID --tag Key=Name,Value=weave-ecs-demo

5. Create the security group and key pair

Create the security group weave-ecs-demo in the VPC as we created a few steps above.

SECURITY_GROUP_ID=$(aws ec2 create-security-group --group-name weave-ecs-demo --vpc-id $VPC_ID --description 'Weave ECS Demo' --query 'GroupId' --output text)

Add inbound rules to the group to allow:

Public SSH access (tcp port 22).
Public HTTP access (tcp port 80).
Private Weave access between instances:
- tcp port 6783 for data.
- udp port 6783 for control in sleeve mode.
- udp port 6784 for control in fastdp mode.
Public and private access to Weave Scope between instances (tcp port 4040) . This is Only needed when not using Scope ‘Cloud’).

aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 80 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 4040 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 6783 --source-group $SECURITY_GROUP_ID
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol udp --port 6783 --source-group $SECURITY_GROUP_ID
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol udp --port 6784 --source-group $SECURITY_GROUP_ID
aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --protocol tcp --port 4040 --source-group $SECURITY_GROUP_ID

Next create a key pair which allows us to access any EC2 instances that are associated with this security group.

aws ec2 create-key-pair --key-name weave-ecs-demo-key --query 'KeyMaterial' --output text > weave-ecs-demo-key.pem

6. Create the IAM role

Create an IAM role for the Weave ECS instances.

aws iam create-role --role-name weave-ecs-role --assume-role-policy-document file://data/weave-ecs-role.json
aws iam put-role-policy --role-name weave-ecs-role --policy-name weave-ecs-policy --policy-document file://data/weave-ecs-policy.json
aws iam create-instance-profile --instance-profile-name weave-ecs-instance-profile
aws iam add-role-to-instance-profile --instance-profile-name weave-ecs-instance-profile --role-name weave-ecs-role

7. Create a launch configuration

Check Weave’s latest ECS AMIs and choose an AMI depending on your configured region.

Set the AMI variable, replacing XXXX with the AMI of your region.

AMI=XXXX

Initialize the instance User Data script.

cp data/set-ecs-cluster-name.sh ./user-data.sh

Optionally (if you would like to use Scope ‘Cloud’ and you are participating in the Early Access program), add your Scope ‘Cloud’ service token to the User Data. Run the following command, replacing XXXX with your Scope ‘Cloud’ service token.

echo "echo SERVICE_TOKEN=XXXX >> /etc/weave/scope.config" >> ./user-data.sh

aws autoscaling create-launch-configuration --image-id $AMI --launch-configuration-name weave-ecs-launch-configuration --key-name weave-ecs-demo-key --security-groups $SECURITY_GROUP_ID --instance-type t2.micro --user-data file://user-data.sh  --iam-instance-profile weave-ecs-instance-profile --associate-public-ip-address --instance-monitoring Enabled=false

8. Create an Auto Scaling Group

Create an Auto Scaling Group using 3 instances available in the first subnet of your region.

aws autoscaling create-auto-scaling-group --auto-scaling-group-name weave-ecs-demo-group --launch-configuration-name weave-ecs-launch-configuration --min-size 3 --max-size 3 --desired-capacity 3 --vpc-zone-identifier $SUBNET_ID

9. Register the task definition

aws ecs register-task-definition --family weave-ecs-demo-task --container-definitions "$(cat data/weave-ecs-demo-containers.json)"

10. Create the demo service

Before launching the demo task, confirm that 3 instances from the Auto Scaling Group have joined the cluster. This should occur shortly after creating the Auto Scaling Group.

To confirm it, run the following command. Its output should show a 3 once all the instances have joined the cluster.

aws ecs describe-clusters --clusters weave-ecs-demo-cluster --query 'clusters[0].registeredContainerInstancesCount' --output text

You will create an ECS Service with 3 tasks, one per EC2 instance in the Auto Scaling Group.

aws ecs create-service --cluster weave-ecs-demo-cluster --service-name  weave-ecs-demo-service --task-definition weave-ecs-demo-task --desired-count 3

Known Issues and Limitations

Auto Scaling Groups are required for ECS to work with Weave. If you create individual instances, they will not work and won’t be able to see each other due to how Weave finds peers in ECS.

For the Advanced User: Build Your Own Weave ECS AMI

If you need to incorporate changes to the Weave AMI, you can do so by following these steps.

Conclusions

You have used Weave out-of-the-box within the Amazon Container Management service or ECS and used Weave for both service discovery and load balancing between containers running in Amazon EC2 instances. In addition to this, you were introduced to Weave Cloud for visualizing and monitoring a Weave container network.

Weave runs regardless of whether it was executed on the same or on different hosts, and can even run across completely different cloud providers if necessary.

You can easily adapt these examples and use them as a templates in your own implementation. We would be very happy to hear any of your thoughts or issues via Help and Support.