Scale test: Running 10,000 Docker Containers on Weave

Welcoming today’s guest blog from the DCHQ team, who use Weave Net to provide the Docker container network for their Deployment Automation and Lifecycle Management platform:Using DCHQ and Weave to automate Docker at...

Welcoming today’s guest blog from the DCHQ team, who use Weave Net to provide the Docker container network for their Deployment Automation and Lifecycle Management platform:

Using DCHQ and Weave to automate Docker at scale

Orchestrating Docker-based application deployments can be a challenge for many DevOps engineers. Pools of servers are often spread among multiple development teams, making deployment difficult to monitor, manage and maintain.

DCHQ provides a deployment automation and life-cycle management platform for docker-based applications. It allows you to define infrastructure provisioning, auto-scaling, and clustering and placement policies for infrastructure operators. Weave provides the networking, enabling container-to-container communication, so that you can maintain and monitor clusters regardless of the host or cloud provider.

This blog is the first in a two part series that address the scalability of clusters with Weave and DCHQ. Part 1 discusses how you can use DCHQ cluster orchestration and Weave container networking to deploy 10,000 NGINX containers to 1,000 clusters in less than 45 minutes.

You will:

• Build a reusable application template for NGINX clusters.
• Automate Virtual Machine (VM) provisioning.
• Automate NGINX cluster deployment using DCHQ’s REST API’s.
• Monitor the CPU, Memory & I/O of the running containers.

To simulate a realistic scenario:

• Create 10 different users on DCHQ.io
• Create 10 clusters, configuring each cluster for 3 cloud servers on Rackspace.
• Assign each user one cluster as the default cluster for application deployment.

Building the Application Template for the NGINX Cluster

Log into DCHQ (either the hosted DCHQ.io or the on-premises version), and create a new Docker Compose template, by selecting Manage > Templates and then clicking the + button.

Next specify the parameters to use for your clusters. For this example, a simple NGINX cluster is created to demonstrate auto-scaling. Set the number of containers to 10 in the cluster_size  field.

Other configurable parameters include:

• mem_min — the minimum amount of memory to allocate for the container.
• host — the host to use for container deployments. Specifying a host ensures high-availability for your server clusters and also allows you to comply with any affinity rules if you are running a database on a separate host.

Creating the Application Template

Provisioning the VMs

After you’ve saved the application template, register with Rackspace, so you can provision the VMs.

Weave is set up in the Network field and is launched after the VMs are provisioned. Weave establishes peer connections between clusters and hosts and afterwards will be ready to automatically discover any newly deployed containers.

10 Rackspace clusters were provisioned with each cluster using 3 virtual machines with 4 GB of memory for a total of 30 VMs.

Creating a Rackspace Account

Rackspace Server Provisioning Details

Specifying Cluster Parameters

 Deploying NGINX Clusters Using DCHQ’s REST API’s

With the servers provisioned, you are now ready to deploy the NGINX container clusters using DCHQ’s REST API’s.

Run the following script to invoke the deployment API (https://dchq.readme.io/docs/deployid).

Deploying Containers

The deploy script does the following:

• Runs through a ‘for loop’, from 1 to 100
• One iteration of the loop deploys a 10-node NGINX container cluster application. In other words, one loop deploys 10×10 (or 100) containers onto 10 clusters.
• user1%40dchq.io is used for user1@dchq.io where @ symbol is replaced by hex %40
• <id> is the URL of the NGINX cluster application ID. The <id> is found by selecting Library > Customize for each NGINX cluster.
• sleep 22 is the time used between each iteration and in this instance, accounts for 2,200 seconds – or 37 minutes.

Monitoring the CPU, Memory & I/O Utilization of the Cluster, Servers & Running Containers

The performance of the VMs was monitored both before and after the 10,000 containers were launched.

Before all 10,000 containers were run, CPU usage was minimal and total memory used was only 16%.

Zero Containers Running: Resources Utilized

With all 10,000 containers running, the memory usage of all VMs, peaked at 74%, which is shown in the screen shots, “All 10,000 Containers Running” and in “Host Details: Resource Utilization”.

All 10,000 Containers Running

Host Details: Resource Utilization

And for comparison this next screenshot displays the memory usage of all 1,000 Nginx clusters, with 10 containers running on each cluster.

All 1,000 Clusters Running

 Conclusions

Orchestrating Docker-based application deployments is still a challenge for many infrastructure operators as they struggle to manage pools of servers across multiple development teams, and where access controls, monitoring, networking, capacity-based placement, auto-scale out policies and quota need to be configured.

By integrating with Weave to provide container-to-container communication across clustered hosts, DCHQ provides advanced infrastructure provisioning, auto-scaling, and clustering and placement policies for infrastructure operators.

Sign Up for FREE on http://DCHQ.io or download DCHQ On-Premises to get access to out-of-box multi-tier Java, Ruby, Python, and PHP application templates along with application lifecycle management functionality like monitoring, container updates, scale in/out and continuous delivery.