How-To do #GitOps with GitHub on Google Cloud Platform using CloudBuild &GKE

Jaroslav Pantsjoha
ITNEXT
Published in
7 min readMar 20, 2020

--

If you are here, then you’ve surely heard about GitOps. Your colleagues rave on about it, perhaps folks in your office whisper it about, or you sat on the bus next to some chap who does ‘use’ GitOps.

Have a read of GitOps Processes and Integration with flux which was recently written by my colleague Sean Rigby which has great coverage of a high-level overview of the topic.

Otherwise, let’s recap the features and dive deeper into components and implementation.

The key features of GitOps implemetnation

  • Strong release consistency and standardisation across environments — You are guaranteed to have your GitHub sourced definition running by design.
  • Strong Audit Trail — PR approved, Git history tracked, great visibility too.
  • Stronger Reliability and Stability — the result of the standardised release process and consistent deployment architecture
  • Improved Developer Experience — Build every commit, any branch, you decide. Configure your Triggers as you see fit
  • Forced Application Quality Ownership — in line with Agile Practices. You build, you Ship.
  • Improved Productivity as a result — more releases, small incremental deliverables at your approval

The technology pre-requisites of the #GitOps release model are Google Kubernetes Engine, Cloud Build, and good ol’ GitHub. There are likely to be implementations of the same with the other cloud providers too.

So you like what you hear. Good, we still have you then. Let’s get you some of that goodness. Let’s recap the epic features and benefits of such a CI/CD approach

The Realised benefits of the GitOps CI/CD pipeline(s)

  • The simplified and automated release pipeline
  • Abstracted the opinionated release process from the developer, and the release team
  • Reduce the self-managed tool maintenance and dependencies
  • Use generic best-practice patterns to carry out releases
  • Reduce ClickOps — it’s a real thing. (You click, click, click some more, then some release happens. Then you verify it Did happen)
  • Enforce the desired remote state of the repository to live

This is the “Get #GitOps done on GCP” guide with “3 easy steps”. I will cover the Component details, as well as deep dive into How-To in the latter section

There are 3 Fundamental Components to #GitOps, used to build up what is a CI/CD model, which comprises of the following:

  1. Continuous Deployment Tool — I use Weaveworks Flux — to pull Release-Env Deployment Repository and apply [all] changes compared to the present state. There are alternatives like ArgoCD of course
  2. The Continuous Integration Componentcomprises of a Code Repository (GitHub) which contains all the application source code as truth, Integrated with CloudBuild, and appropriate triggers.
  3. The CI/CD Release Strategy, The GitOps Logic, to perform releases for aforementioned CI builds, for CD component to pickup automatically

The 3 “Easy” Steps to getting GitOps CI/CD Model

1 — Configure The Continuous Deployment

I use Weaveworks flux — This step Configure & Deploy Flux on GKE. The detailed how-to is already blogged by Sean Rigby

Apply the whole list of Yaml Manifests, the very Deployment, Account, Memcached Deployment & Service and The secret will feature the github <manifests-deployment-repository>:branch to pull and kubectl apply -f . manifests from.

2 — Integrate Source Repository with CloudBuild

The following cloud repositories are supported with CloudBuild integration: GitHub, BitBucket and Cloud Repositories (Google’s own offering)

The Process is relatively simple. Follow it through and select the application repository which needs to be connected.

Next, you need to configure that bit with a trigger to “Do Stuff” on Merge/Commit. This is where you configure your Trigger.

I have configured my trigger to use cloudbuild.yaml for a particular branch: master events.

You can configure your Trigger to CloudbuildDockerfile and automatically build when you introduce changes for example. The reason we specify the cloudbuild.yaml instead — as the name suggests — this allows us to define more scope for the CloudBuild to do.

This is it. Each time you introduce a change to the master branch, CloudBuild will perform a run, executing steps within that cloudbuild.yaml definition. What is inside the file you say? Well, that gem is for the next section of #GitOps.

3 — CI/CD Release Strategy — The GitOps logic

The foundation here is that this GitOps logic will operate as just yet another cloudbuild.yaml featured build step for the application.

It will run as a separate image hosting that “logic script of GitOps” of operations that we wish to perform. What are those operations, how do we access yet another <deployment-repository> despite building from the <application-repository> All Good questions. This is what we need to incorporate into the gitops image.

  • Decrypt SSH Key Secret for the Flux(CD component) operated GitHub <deployment-repository>
  • Update the Deployment Application Build Hash for the developmentbranch of the <deployment-repository>
  • Optional Bonus: Create Merge Pull Requests for the same Application Build Hash for Staging/Production branches of the <deployment-repository>

Additional Release strategy to consider: Trigger for Development Branch(es). Food for thought. Many ways to skin this cat. Let me know your thoughts in the comments section below.

Meanwhile, let’s get a visual on this:

<application-repository> master branch Commit/Merge event(s) Trigger a build of the specified cloudbuild.yaml

Your Typical Multi-step build — you get those in Gitlab, Jenkins, circle-ci pipelines and the like. Hopefully straight forward

I still have you with me? Good.
The arguments are bash parameters that are there to make this gitops image more interoperable between application. PS you would never use the latest tag. This is just a simple, PoC example. Stay with me.

What are these arguments and how they tie into the GitOps logic? We’re getting there. Here are the screenshots below for the reference.

The Dockerfile reference to build such gitOps image. Taken me an afternoon to polish, perhaps longer. It’s largely a vanilla OS, with google toolkit to enable decryption of the <deployment-repository> ssh key, sentimental option, I know. It also features the sed, git, and hub — the latter allows the Pull Request creation. Handy that.

This demo showcases the use of KMS encrypted SSH-Keys. Encoded blob of which makes way onto gitops docker image
Fundamental and basic logic. Use incoming parameters to process through your own strategy. See example for development environment string matching, to automatically commit&push the unprotected development <depeloyment> repostoy, containing the application’s YAML manifests

Needless to say, each such respective GKE Cluster(DEV, STAGE, PROD) will run it’s own flux deployment, which in turn — will poll the respective-env branch(or repo) of <deployment-repository> containing the YAML manifests, on GitHub.

So This GitOps thing — sounds useful

The GitOps CI/CD model complements the Agile practices and reinforces the DevOps methodology, to enabling frequent incremental releases across Environments.

Core Benefits of Lean Mean GitOps CI/CD Machine:

- Mean Time To Market — Reduce the Lead Time, from the commit of code to code running in production.

- Observability and Monitoring — Consistent and simplified release model to increase the feedback from the earliest stages of the development cycle, through to environments into production. “What is the GitHub repository is what is running in production”

- Continuous Integration and Innovation — simpler release process is easier to understand, review and optimize. Infrastructure-as-Code approach improves portability across cloud providers

- Cost Of Maintenance — reduce, eliminate ongoing cost of toil to configure, patch, fix, redo effort when engineering a bespoke release solution or locked into alternative CI/CD platform.

  • Improve security — With reduced patching, it reduces human error and vulnerabilities within the CICD operating environment. Full GitHub purposed log trails via changeset history

Famous, last, words, of wisdom.
There is a certain amount of freedom to engineer-what-you-like. Just be certain that it’s not turning into a Pet project of your own, and your’s own only. Peers must be able to understand logic through and through to review troubleshoot and debug if necessary. If there is a full-time, permanent team supporting such gitops release strategy design, — super.

In short, Keep your own gitOps logic simple, lean, peer-reviewed, and with associated quality architecture design diagrams (borrow mine if need be) for documentation.

An open-source project maintained solution in this area is most likely best, particularly if well maintained. And I would be happy to trial and consider such OSS, when I find one suitable. If you know one, then please do leave me a message in the comments below. I would be keen to explore such an option(s) in the next implementation of GitOps.

Enjoyed? Please like the post and do share it along!

PS There is quite a number of exciting Kubernetes projects taking place at Contino. If you are looking to work on the latest-greatest infrastructure stack or looking for a challenge, — Get in touch! We’re hiring, looking for bright minds at every level. At Contino, we pride ourselves on delivering the best practices cloud transformation projects, for medium-sized businesses to large enterprises.

JP

By the way, 👏🏻 *clap* 👏🏻 your hands (up to 50x) if you enjoyed this post. It encourages me to keep writing and help other people finding it :)

--

--