Photo by Daniel Cheung on Unsplash

On automating everything — Cloud Container Provisioning on AWS

Let’s talk about applying continuous deployment and integration (CI/CD) to put some services online — the GitOps way. In a nutshell, we like to commit new features to Git, and have these features pushed to users immediately via micro-services. The command-line for Developers looks like this:

git commit -am "add new feature X for product Y" \
  && git push

After typing the line above, a pipeline starts, things get tested and (re-)deployed (blue-green style), after which end-users consume the updated service. Version rollbacks simply work by reverting to a previous version in Git, the service(s) change back accordingly.

This article is mostly about putting the automation parts in place before applying the infra-as-code itself — serving this never-ending itch of finding a faster way to deploy platforms. At the end we create a platform that converts Dockerfiles to running services on AWS Fargate.

Low Maintenance Ops

Modern day infra work involves serving and managing countless platforms, instant delivery on request. That means — yet more — automation.

We start by complementing this Docker & Makefile part to push infra-as-code to public cloud platforms. These are the one-liners that should eventually do the job:

# Make-Git: create a build repository
make git url=${GIT_URL_NEW_PLATFORM}
# Make-Infra: deploy the infrastructure
make infra

Excluding commentary, that is two lines of code. The first — Make-Git — is there to pull code parts in a new deployment object. The second — Make-Infra — handles the deployment itself.

Note that all code is available on Github. Links to the repositories are provided in the final section of this article (Dockerfile to Fargate).

Make-Git

Make-Git facilitates re-using existing code pieces in a new project. The tool is a wrapper around GIT -subtrees and -submodules. We maintain individual code parts in one spot, and pull in when needed.

Figure 1: Make-Git

Make-Git (Figure 1) pulls in existing — re-usable — parts into a new platform, based on a configuration in .gitmodules. In our example we create two re-usable parts, Elastic Container Service (ECS) and Code Pipeline.

Make-Git is a Python module, installed through PyPi:

python3 -m pip install makegit

There is no need to copy this line. The module is added by Dockerfile, demonstrated in the final chapter of this article. Entry in Makefile:

git:
     $(RUN_DOCK) "GIT_URL=\"$(URL)\" makegit"

That git target triggers this ~/.aliases function in container runtime:

# compressed for readability purposes
function __makegit(){
    ...
    python3 -m makegit --url "${GIT_URL}"
    ...
    return $return_code
}
alias makegit='__makegit'

The setup allows for running the Python3 module inside the container, from where it collects the required GIT repositories.

By using the ~/.aliases file we can add code before and after, and even use special characters — quotes, pipes and redirection — that are notoriously difficult to escape in Makefile itself.

Deploy infra by Commit — VIM users bonus

I grew accustomed to commit & push code continuously — every few to 10 minutes — to get immediate feedback on tiny increments. If you are a VIM user, you may like this ~/.vimrc snippet:

Type :P — sorry, no pun intended — to commit and push code, or :W to only commit. This is for VIM, but every IDE should be capable of running a similar function — feel free to copy from the ~/.vimrc snippet selectively.

Make-Infra

The infra-as-code pieces are written in Terraform as I had the code readily available — it required limited contribution on my part. Good arguments to use Terraform are:

• Integrates: one code base to connect to all relevant cloud-providers.
• Documented: available blueprints help to kick-start a new project.
• Configurable Remote State: this helps working on a shared project.

Disclaimer. If you use AWS only, CloudFormation is arguably a better choice — its automatic rollbacks and deep AWS integration are golden. I will share the CloudFormation version in a next article, in this article I stick with Terraform.

In a production setting, Terraform requires setting up and retrieving a state first, before running the actual deployment code. Thanks to some Python automation and Makefile magic, we compressed it all into one call.

Figure 2: Make-Infra

Make-Infra schematics in Figure 2. Via Makefile the Python module RemoteState is called to setup Terraform state material (1), followed by plain Terraform build commands to deploy the infra (2).

On Terraform RemoteState

In an enterprise environment, Terraform is typically used by one or more teams, to ensure safe operations Remote State & Locking is applied.

• Remote State. Infra details, such as service IDs and endpoints, are written in a state-file by Terraform. Storing this file in a remote location, provides a single source of truth on deployed infrastructure — stored in S3.
• Locking. Only one update process can run at a time. We enforce this by setting a lock entry in a database — stored in DynamoDB.

One common pain, is that an incorrect backend file can easily cause a faulty state, which is painful to troubleshoot and recover from. To solve that problem, the RemoteState module creates this file automatically.

On RemoteState — Module

The RemoteState module creates:

• DynamoDB and S3 parts on AWS, if the parts do not yet exist.
• A backend file for Terraform to use — terraform/backend-auto.tf

The module derives information from the current GIT repository — pulled by Make-Git. AWS keys and region information are loaded from the default ~/.aws/credentials and ~/.aws/config files (see AWS CLI instructions).

This installs the RemoteState module:

python3 -m pip install remotestate

Similar to Make-Git, this one-liner is included in a Dockerfile. Entry in Makefile:

remote:
        $(RUN_DOCK) "remotestate"

The wrapper function in ~/.aliases:

# compressed for readability purposes
function __remotestate(){
    ...
    cd "${PROJECT_PATH}" && \
        python3 -m remotestate --git "build/buildrepo"
    ...
    return $return_code
}
alias remotestate='__remotestate'

RemoteState expects a GIT repository in the “build/buildrepo” directory, created by Make-Git in earlier chapter, to retrieve the repository URL defined in .git/config. From the URL, RemoteState derives the names for the DynamoDB tables and S3-Bucket automatically, in the next section we explain how this works

RemoteState: Naming Scheme

RemoteState takes care of naming DynamoDB tables and S3 Bucket, based on the input of a GIT repository URL. Assume the following URL:

GIT_URL = https://github.com/LINKIT-Group/aws-cicd-demo

This can be decomposed in to the following parts:

# composed from ${GIT_URL}
GIT_HOST = github.com
GIT_ORGANISATION = LINKIT-Group
GIT_REPO_NAME = aws-cicd-demo
GIT_URI = ${GIT_HOST}/${GIT_ORGANISATION}/${GIT_REPO_NAME}
# defaults -- optionally retrieved from ${GIT_URL}
GIT_BRANCH = master

This information is used to create names for DynamoDB and S3. All characters are lower-cased:

# note that one set of DynamoDB tables is created to hold all repositories within one group (GIT_ORGANISATION).
# DynamoDB Table base name -- not a Table itself
DYNAMO_TABLE_NAME = terraform.${GIT_HOST}.${GIT_ORGANISATION}
# table for locking
DYNAMO_TABLE_NAME_LOCK = ${DYNAMO_TABLE_NAME}.lock
# table to store key=${GIT_URI}, value=${S3_BUCKET_NAME} -pairs
DYNAMO_TABLE_NAME_LOCK = ${DYNAMO_TABLE_NAME}.s3
# Python Code example
import time
def time_string():
   """Return current time in micro-second (usec) as a string"""
   return str(round(time.time()*10**6))
CREATION_TIME_MICROSEC = time_string()
# Bucketname must be globally unique--why we add the microseconds.
S3_BUCKET_NAME = ${DYNAMO_TABLE_NAME}-${CREATION_TIME_MICROSEC}

Troubleshoot Terraform

To run Terraform, these are the lines in Makefile:

infra:
        $(RUN_DOCK) "remotestate"
        $(RUN_DOCK) "terraform init \
        && terraform plan \
        && terraform apply -auto-approve \
        && terraform output"
destroy:
        $(RUN_DOCK) "terraform destroy -auto-approve"

That is nice, but how about all the other Terraform commands to manage or troubleshoot deployed infra? Enter the container as follows:

make shell

After entering the container, you can directly type individual Terraform commands to troubleshoot and fix the issue.

Low Maintenance Ops: Wrap-up

A team member can now pull the GIT repository by applying Make-Git, and create an infrastructure using Make-Infra.

The main advantage in both of these processes is that team members do not have to deal with internal configuration, speeding up delivery while reducing risk. Needless to say, all is automated.

Dockerfile to Fargate

The previous sections were about creating and deploying new platforms efficiently. In this part it’s time to deliver the promised CI/CD platform.

Additional Prerequisites

• An AWS account — this can be acquired for free.
• Docker. Install instructions: Ubuntu or Mac.
• Ability to run “make” on your host system. Most Unix, Linux and Mac systems have this by default — on Windows I’d recommend an Ubuntu VM.

I recommend using a clean AWS account or a dedicated test region to play in. I am also assuming you have some experience using AWS and understand what we are building.

A build pipeline on AWS for server-less containers

Figure 3: CI/CD + ECS in AWS

Assuming your system is ready — see list of prerequisites above — the following lines deploy this platform:

# retrieve deploytools, this repository holds MakeGit and MakeInfra
git clone https://github.com/LINKIT-Group/deploytools
cd deploytools
# collect the code to deploy our demo
# code is put in the build/ directory
make git url=https://github.com/linkit-group/aws-cicd-demo
# deploy the demo
# note: ensure ~/.aws is setup properly!
make infra

Building the infrastructure takes about 5 minutes — just enough time to grab a cup of coffee, tea or a snack. If all goes well, you should see this Terraform output on your console:

# Apply complete! Resources: 47 added, 0 changed, 0 destroyed.

Do note the AWS bill is now running. Approximately a few dollars a day.

Now it’s time to access the app. Login to AWS console and go to EC2 -> Loadbalancers, here you will find an application loadbalancer: “app-demo-backend-test-alb”. Click on it, open the “Description” tab, copy the value under “DNS name”, and paste it in a browser window.

You should get this error message back from the application loadbalancer:

503 Service Temporarily Unavailable

You get this error message because there is no application yet. We only created the build platform, waiting for us to push application code.

Pushing application code

Time to switch hats with a developer. Let’s install an Nginx container and say hello to the world. We clone this repository, copy/ paste the contents to the CodeCommit repository, and deploy via commit and push.

Retrieve the ${CODECOMMIT_URL} by browsing to AWS console -> CodeCommit. There should be a repository named “app-${random-chars}”, click it and go to “Clone URL”. Choose “Clone SSH” and copy the string.

If this is your first time to clone a CodeCommit repository, you get an access denied error. First, setup an SSH key following these instructions (Step 3).

# clone CodeCommit repository -- first time its empty
git clone ${CODECOMMIT_URL}
# clone nginx demo code
git clone https://github.com/LINKIT-Group/nginx-docker-helloworld
# copy/ paste to CodeCommit repo
cp nginx-docker-helloworld/* ${CODECOMMIT_REPO}/
# commit and push -- this will trigger a container deployment
cd ${CODECOMMIT_REPO}
git add *
git commit -am "add feature X for product Y"
git push

Now it’s time to sit back and relax. You can view the process by browsing to AWS console -> CodePipeLine, and refresh the URL pointing to the container.

If you get the “Hello world” message, it works as designed and we are done.

# delete the platform
make destroy
# note this will not clean up the S3 bucket and DynamoDB table used by Terraform backend. These must be deleted separately if no longer needed.

The End of the Beginning

I hope to have proven that you can stop using EC2-instances for simple services. Did you notice ECS-Fargate does not insert EC2 either? — these are server-less containers.

Next step is to see how far we can scale this up, and start saving time on managing big container clusters — no more clunky EC2-instances!

Another work-item for me is to replace all the Terraform code with pure CloudFormation and switch to Lambda-only for the automation glue.

Hope to share either (or both) in a future article.

Thanks for reading!