Lessons Learned: Switching from CircleCI to Google Cloud Build

Earlier this month we finally hit CircleCI’s generous 1500 minute per month build time limit.

As the founder of a bootstrapped startup (see Focuster) I’m super grateful for CircleCI over the years because it has been a great build service and they were very generous with their limits.

But every new expense feels like a punch to face.

We recently migrated Focuster’s infrastructure over to Google’s Kubernetes service (GKE) recently and I noticed they had their own build service, Cloud Build (GCB).

Though it wasn’t as comprehensive in many ways, it had some clear advantages for me at this point.

I wanted to share what I learned in the migration so that others might benefit and hopefully Google will listen to some of my feedback and make some improvements.

Pros of Google Cloud Build

Less Expensive *

As I mentioned earlier, CircleCI provides 1500 of minutes per month on their free tier with a concurrency limit of 1. To move to the next level of service is a flat rate of $50/mo. for 2 build concurrency with unlimited builds.

Google Cloud Build on the other hand provides a free tier of 120 build minutes per day on their standard size container which is an n1-standard-1 machine type (1 core, 3.75gb RAM).

This works out to approx. 3600 build/minutes per month. That said the time for unused days doesn’t roll over so a particular heavy day may incur some charges.

Above the free tier the pricing is only $0.003 per build/minute. My builds are averaging about 10 minutes so I’d have to do over 1667 builds a month, or 50 builds a day to hit the $50 that CircleCI charges.

Concurrency

CircleCI basically charges you based on your concurrency level and gives you unlimited minutes. GCB has a practical limit of 10 concurrent builds but charges you per build/minute. I like this approach at this point because I can get the benefits of the extra concurrency while on the free tier.

Local testing

Both CircleCI and GCB provide tools to test your deploys locally using docker. Last time I tried this with CircleCI on OSX it was horribly slow to the point where it was unusable. I’m not sure if that problem was ever resolved or not.

GCB provides tools for local testing that are distributed with the Google Cloud SDK via the cloud-build-local command.

Using this tool I was able to iterate on my build steps quickly without using up any time on the cloud service.

Faster deploys

• Proximity to Google Container Registry (GCR) for faster docker pull/push.

I didn’t actually test that this is true or not but I assumed that being in Google’s network meant that pulling a docker image from GCR to GCB would be faster than from CircleCI.

• Real-time Image Notification deployment notifications

For deployment to my Kubernetes cluster I use a tool called Keel to automatically update my deployments when the image in the Docker repository is updated.

In the past I used a polling system with Docker because the web hooks were a bit of a pain to setup. I found setup of the GCR PubSub notifications to be really easy and this made deploys up to a minute faster.

Docker Native Steps

Both CircleCI and Google Cloud Build have extensive Docker support. But Google Cloud Build lets you define your own steps using Docker images.

A shared workspace (mounted at /workspace) is persisted between steps.

I found this actually a pretty nice way to develop my build steps because I could leverage small purpose-built images for each step.

I could also leverage existing Docker images at any point in the build process and I can write my build steps in any language I want.

Cons of Google Cloud Build

The builtin steps are limited

GCB provides existing steps for many of the most common tools: docker, git, Google’s services, curl, npm, kubectl, etc.

Complete list here: https://github.com/GoogleCloudPlatform/cloud-builders

But for doing something as simple as initializing git submodules was difficult. I had to create my own build step for that. It would be great if they provided an easy to use build step that would pull the ssh key out of Google’s KMS.

CircleCI has the advantage here as they support adding SSH keys to the build account and because their build is all running in a single container its much easier to configure.

Poor caching story

Another area where CircleCI shines compared to GCB is in the caching story.

They provide some caching primitives that are quite flexible in terms of dynamically generating a cache key based on the hash of a file’s contents for example. And allowing fallback to an older cache based on the branch name or many other criteria.

GCB on the other hand only provided documentation on storing some folder contents in Google Storage.

The good news, I was able to create my own build step that essentially mimicked CircleCI’s behavior in GCB.

(I will open source the caching steps I created if there’s interest)

Limited debugging

CircleCI makes it really easy to SSH into your build to diagnose a problem by simply checking off a box on the build and restarting it.

With GCB I couldn’t find any comparable step.

It might be possible to use docker run in local debugging to open a shell in a particular image but it’s not clear to me how to preserve the workspace of the build in this case.

The UI isn’t as nice

• Filter builds could use much improvement. Full text search of all the build attributes would be great. There’s no explanation of the query language.
• Can’t see progress of individual steps until the build is finished or need to wade into the logs for the whole build. Circle CI does a much better job here as you can literally see the progress on each step.
• You have to manually click refresh to see new builds in the build list.

Lacking integrations

Setting up Slack and Github integration is based on manual installation of some Google Cloud Functions, rather than a 1-click that most services use.

That said, the use of the PubSub for build notifications allows for a lot of customization.

Limited Machine Type Configurations

Right now you can only choose between 1 core, 8 core or 32 cores. Would be nice to have 2 and 4 cores as options. Could we get a n1-standard-2 or other sizes for build machines?

Final Thoughts

Having made the switch I am overall pretty happy with the results. I hope that Google keeps making regular updates to this service.

—

This is part of a series of articles about Focuster’s engineering practices.

Check out Focuster, the first focus app that actually helps you get things done instead of making endless lists. Automatically build a schedule from your to-do list in Google Calendar, and move things forward if they don’t get done.

Follow @focusterapp on Twitter to get updates on productivity and high performance.