12 factor Microservice applications — on Kubernetes

The 12-factor methodology has been widely adopted for building modern, scalable, maintainable applications. While there is a wealth of knowledge available on these principles, what’s often missing is practical guidance on implementing them in the context of microservices. This article aims to bridge that gap. We’ll explore how to utilize the 12-factor methodology for designing microservices, the tooling choices, and how to bootstrap a 12-factor microservice application(s) on Kubernetes.

Challenges and Goals

Our goal is straightforward yet challenging: we aim to bootstrap a 12-factor microservice application(s) with a dashboard, and identify the leanest toolkit required for the task. The selected tools should align with the principles of the 12-factor methodology.

First Factor: Codebase

One codebase tracked in revision control, many deploys.

The first factor of the 12-factor methodology dictates that there should be “one codebase tracked in revision control, with many deploys”.

To manage our Codebase we need a VCS! Version control systems (VCS) are integral to any modern software development process. They help teams manage changes to source code over time, allowing multiple people to collaborate, track modifications, and revert changes when necessary.

When it comes to choosing the right VCS, there are several factors to consider, primary among them:

• Branching: is a core feature of a version control system that allows developers to diverge from the main line of development and create isolated environments to work on features or fix bugs. Each branch represents an independent line of development that does not affect other branches, allowing many developers to work simultaneously on different features.
• Actions and Workflows: Branching becomes even more powerful when combined with a strategy, like Actions and/or Workflows. These strategies define a set of rules on how and which step functions are executed, when a codebase is committed.
• Other factors: Ease of use, community and support, and integrations.

For this purpose, we’re going to use Git(Hub), coupled with Git Actions and Git Workflows.

The Role of Git in Our Design

In our design, both backend and frontend developers will utilize Git, and stretch it.

Backend developers will write the business logic within application(s), which can be coded in any preferred language. The development process will involve

• scaffolding the application on a “feature” environment,
• writing tests first (a practice referred to as Test-Driven Development or TDD),
• writing the business logic (BL), and
• finally submitting a Pull Request (PR).

Frontend developers will follow a similar pattern, writing frontend business logic using APIs served by the application. Like the backend, frontend applications can be coded in any language, require scaffolding, and follow similar branch and promotion criteria.

We’ll create a Git repository for each microservice and front-end application(s). This allows us to use branches like development, staging, and main (or feature branches, if necessary) for our development topology. Each microservice will have “deployment manifests” in addition to the actual source code. These manifests are necessary for deploying the application(s) to Kubernetes.

Docker and a Docker Registry

A necessary diversion here to mention Docker and it’s Registry for hosting application images!

Docker is an open-source platform designed to automate the deployment, scaling, and management of applications. It uses containerization to encapsulate applications and their dependencies into a standardized unit for software development and ensures that the application runs seamlessly in any environment, whether it’s a local machine, a private data center, or the public cloud.

Docker accomplishes this by using a lightweight container runtime that isolates the application’s processes from the rest of the system. Each Docker container runs independently and includes everything it needs to run the application: code, runtime, system tools, libraries, and settings.

A Docker Registry is a storage and distribution system for named Docker images. These images are the building blocks of Docker containers. They contain the application, its dependencies, and some metadata describing what the container should do when it’s run.

Key features of any Docker Registry include:

• Version control: Every time you push an image to the registry, it creates a new version. This allows you to roll back to an earlier version if necessary.
• Access control: You can control who has access to read from or write to your registry, and what they can do.
• Integration: Docker registries can be integrated with your CI/CD pipeline, ensuring that the images used for deployment are always up to date.

If Docker images are the means for packaging software, then the next natural question is how will the docker images be deployed? How will they run in multiple envs; like stage, qa, production?

Without a way to deploy the docker images (to run them as containers) the images themselves are useless. So, what does it mean by “a way to deploy the docker images”?

Requirements for “A way to deploy docker images”

• a way to specify where to run the docker images — a set of virtual machines on which the docker images will be deployed
• a way to specify how to run the docker images — the resources required to run the image (ex. Memory/CPU/GPU, disk, etc.)
• a way to route traffic to the docker container (nginx-ingress)
• a way to monitor if the docker container is healthy (health checks)
• a way to restart the docker container if it fails/crashes (rollout, restarts)
• a way to maintain a desired set of replicas of the container

All of the above features are bare minimum for the deployment of docker images successfully. This “way to deploy docker images” is called Container Orchestration, and we use Kubernetes for all this.

Embracing Kubernetes

In our design, we’ve chosen Kubernetes as the orchestration platform for our microservices. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications. It provides a platform to run distributed systems resiliently, scaling and recovering applications as needed.

Declarative Instructions

Our application manifests are declarative YAML files that contain all the useful information about our microservices, including:

• Application metadata (name, port): Helps identify the specific microservice from the pool.
• Specs (replicas, image, resource limits, etc.): Provides sizing and scaling information.
• Ingress rules: Establishes routing information to direct incoming requests to the correct API or frontend application.

We’ll host our application manifests in a separate Git repository. This approach allows us to link the corresponding infrastructure repository with the repository containing the actual source code, providing a clean separation of concerns.

Our directory structure for the customizable manifests will look like this:

~/manifests-index develop > tree -L 1 infra/k8s   
infra/k8s  
├── auth  
├── auth-mongo  
├── ingress  
├── kafka  
├── posts  
└── posts-mongo

A Closer Look at Manifests

Inside each of the folders designated for each microservice (or its associated database), we’ll keep customizable manifests for all our named environments with base specifications and overlays for development, staging, or any other (say feature).

Design Decision: Kustomize to build manifests

We’ve chosen to use Kustomize to “build” our manifests, which offers a template-free way to customize application configuration. It allows us to create a base set of resources and apply a set of customizations as needed for each environment.

Here’s what our Kustomize directory structure will look like:

~/manifests-index develop > tree -L 4 infra/k8s/auth  
infra/k8s/auth  
├── README.md  
├── base  
│   ├── deployment.yaml  
│   ├── kustomization.yaml  
│   └── service.yaml  
└── overlays  
    └── development  
        ├── build-version-patch.yaml  
        └── kustomization.yaml

And the deployment, service and applied kustomization!

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: auth
  template:
    metadata:
      labels:
        app: auth
    spec:
      containers:
        - name: auth
          image: xx-xxx-docker.pkg.dev/labs-xxxxx/auth/auth:development
          env: 
            - name: API_VERSION
              value: 'v1'
            - name: MONGO_URI
              value: 'mongodb://auth-mongo-srv:27017/auth'
            - name: JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: jwt-secret
                  key: JWT_KEY  

apiVersion: v1
kind: Service
metadata:
  name: auth-srv
spec:
  selector:
    app: auth
  ports:
  - name: auth
    protocol: TCP
    port: 3000
    targetPort: 3000

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: development

patchesStrategicMerge:
  - build-version-patch.yaml

In affect kustomize build should fetch us all the deployment manifests for a microservice, for a named target environment.

kustomize build infra/k8s/<microservice>/overlays/<overlay>

kustomize build infra/k8s/auth/overlays/development

Why is there a build-version-patch.yaml ? More on it later …

Why Ingress and Kafka Manifests?

We treat ingress as an application to provide fine-grained routing control, allowing us to customize routing rules for any given branch or namespace.

Treating Kafka as an application allows us to boot up an instance of Kafka with all its topics per namespace (using Strimzi, will do a post on it). This approach provides a scalable, distributed event streaming platform that our microservices can use for communication. However, note that hosting Kafka per namespace can get expensive, so larger teams may prefer to use a central development Kafka instance instead.

Ingress and it’s base!

~/_workspace/manifests-index develop > tree -L 3  infra/k8s/ingress 
infra/k8s/ingress
├── README.md
├── argo
│   ├── base
│   │   ├── application.yaml
│   │   └── kustomization.yaml
│   └── overlays
│       └── development
├── base
│   ├── ingress-srv.yaml
│   └── kustomization.yaml
└── overlays
    └── development
        └── kustomization.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: dev.xxxx.io
      http:
        paths:
          - path: /api/v1/users/?(.*)
            pathType: Prefix
            backend:
              service:
                name: auth-srv
                port:
                  number: 3000
          - path: /api/v1/posts/?(.*)
            pathType: Prefix
            backend:
              service:
                name: posts-srv
                port:
                  number: 3000
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: dashboard
                port:
                  number: 80

Kafka and it’s base!

~/_workspace/manifests-index develop > tree -L 3  infra/k8s/kafka
infra/k8s/kafka
├── argo
│   ├── README.md
│   ├── base
│   │   ├── application.yaml
│   │   └── kustomization.yaml
│   └── overlays
│       └── development
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── topics.yaml
└── overlays
    └── development
        └── kustomization.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.3.1
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.3"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Since we are using Strimzi as our Kafka orchestrator per namespace, a benefit accrued is that we can have unique topics per developer requirement!

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: comment
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  partitions: 1
  replicas: 1
  config:
    retention.ms: 900000
    segment.bytes: 1073741824
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: post
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  partitions: 1
  replicas: 1
  config:
    retention.ms: 900000
    segment.bytes: 1073741824

Index Repository Overview

Here is what it looks like …

~/manifests-index develop > tree -L 1
.
├── README.md
├── auth
├── infra
├── npm-commons
├── posts
├── frontend
├── .gitignore
└── skaffold.yaml

• The auth and posts directories are dedicated repositories for the authentication and posts microservices.
• npm-commons is a node commons library with shared middlewares, events, errors among others.
• there are frontend applications
• and there’s scaffolding instructions in skaffold.yaml
• a .gitignore which excludes all except the infra and skaffold

Can do a detailed post on this section, if enough votes!

Scaffolding with Skaffold

In our design, developers (whether backend or frontend) will use Skaffold to bootstrap an environment for their work. Skaffold is a command-line tool that facilitates continuous development for Kubernetes applications.

Here’s an example of our Skaffold configuration:

apiVersion: skaffold/v2beta29
kind: Config
deploy:
  kubectl: 
    manifests:
      - ./infra/k8s/auth/overlays/"{{.IMAGE_TAG}}"
      - ./infra/k8s/posts/overlays/"{{.IMAGE_TAG}}"
build:
    tagPolicy:
      envTemplate:
        template: "{{.IMAGE_TAG}}"
  artifacts:
    - image: xx-xxx-docker.pkg.dev/labs-xxxxx/auth/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
       manual:
         - dest: .
           src: 'src/**/*.ts'
    - image: xx-xxx-docker.pkg.dev/labs-xxxxx/posts/posts
      context: posts
      docker:
        dockerfile: Dockerfile
      sync:
       manual:
         - dest: .
           src: 'src/**/*.ts'

The IMAGE_TAG environment variable, helps skaffold identify the namespace to deploy to.

Let’s create a namespace eg:development , and export an IMAGE_TAG environment variable also as development.

kubectl create namespace development
export IMAGE_TAG=development

From here on a simple skaffold dev will bring up the development environment.

> skaffold dev

Git Workflow and IAM Considerations

One of our Git Workflows will include steps for

• cloning the manifests repository,
• substituting variables in our manifest files, and
• commit to the index reconciliation repository.

We will also include IAM considerations and use organization repository secrets for managing sensitive data.

  commit:
    name: Commit to Reconciliation repository
    needs: [build-and-push]
    runs-on: ubuntu-latest
    if: ${{ needs.build-and-push.result == 'success' }}
    steps:
      - name: Echo
        id: echo-sha
        run: |
          echo "github_sha: " $GITHUB_SHA 
          echo "::set-output name=sha_short::${GITHUB_SHA::7}"
      - name: Clone
        run: |
          git config --global user.email ${{ secrets.GH_USER_EMAIL }}
          git config --global user.name ${{ secrets.GH_USER_NAME }}
          git clone -b develop https://.:${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}@github.com/<your-org>/manifests-index.git
          echo "clone repo success"
      - name: Substitute
        run: |
          sed -i "s/value.*$/value\: $GITHUB_SHA/"  manifests-index/infra/k8s/<microservice>/overlays/development/deploy-version-patch.yaml
      - name: Commit
        run: |
          cd manifests-index && git add . && git commit -m '${{ github.event.repository.name }} updated with ${{ steps.echo-sha.outputs.sha_short }} on branch ${{ github.head_ref }}' && git push origin develop

Another workflow will build and push docker images!

env:
  PROJECT_ID: ${{ secrets.GKE_PROJECT }}
  REGION: ${{ secrets.GKE_REGION }}
  IMAGE_NAME: auth #Rule: same as repository name
  IMAGE_TAG: development #${{ github.sha }} #Rule: same as branch name
  GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} # reqd for update docker image to ggl Artifact Repository

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    # Add "id-token" with the intended permissions.
    permissions:
      contents: 'read'
      id-token: 'write'
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v3
    
    # Authentication for gcloud Artifact Registry via credentials json
    - id: 'auth'
      uses: 'google-github-actions/auth@v1'
      with:
        credentials_json: '${{ secrets.GCP_SA_KEY }}'

    # Setup gcloud
    - name: Set up Google Cloud SDK
      uses: google-github-actions/setup-gcloud@v1
      with:
        project_id: ${{ env.PROJECT_ID }}
        service_account_key: ${{ env.GCP_SA_KEY }}
        export_default_credentials: true

    - name: Configure Docker
      run: |
        gcloud auth configure-docker
    
    - name: Configure Artifact Registry
      run: |
        gcloud auth configure-docker \
        ${{ env.REGION }}-docker.pkg.dev

    - name: Build and push Docker
      uses: docker/build-push-action@v2
      with:
        push: true
        tags: ${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.IMAGE_NAME }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}

Application Reconciliation and Achieving “One Codebase, Many Deploys”

Why is there a build-version-patch.yaml ?

With every successful build of any application, we will patch the build version on the named overlay, triggering application reconciliation. This process ensures that the desired state of our application defined in the Git repository matches the actual state in our environment. It plays a significant role in achieving the “one codebase, many deploys” principle of the 12-factor methodology.

- name: Substitute
  run: |
    sed -i "s/value.*$/value\: $GITHUB_SHA/"  manifests-index/infra/k8s/<microservice>/overlays/<overlay>/deploy-version-patch.yaml

Administration and ArgoCD

ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, will also play a crucial part in our setup. We will install and configure ArgoCD as per the official documentation and create an application for each of our microservices in ArgoCD.

ArgoCD configuration will look like this, eg auth:

apiVersion: argoproj.io/v1alpha1  
kind: Application  
metadata:  
  name: auth  
spec:  
  destination:  
    namespace: development  
    server: https://kubernetes.default.svc  
  source:  
    repoURL: https://github.com/<your-org>/manifests-index.git  
    targetRevision: develop  
    path: infra/k8s/auth/overlays/development  
  project: default  
  syncPolicy:  
    automated: {}

And this manifest too can be included in our index

~/_workspace/manifests-index develop > tree -L 4 infra/k8s/auth  
infra/k8s/auth  
├── README.md  
├── argo  
│   ├── README.md  
│   ├── base  
│   │   ├── application.yaml  
│   │   └── kustomization.yaml  
│   └── overlays  
│       └── development  
│           └── kustomization.yaml  
├── base  
│   ├── deployment.yaml  
│   ├── kustomization.yaml  
│   └── service.yaml  
└── overlays  
    └── development  
        ├── build-version-patch.yaml  
        └── kustomization.yaml

And from now on, an argo specific kustomize build will generate the argo manifest for an application. Automate everything!

kustomize build infra/k8s/<microservice>/argo/overlays/<overlay>

kustomize build infra/k8s/auth/argo/overlays/development

Note that FluxCD is a powerful alternative!

Templating for Automation

It’s a good practice to templatize the primary manifests, to facilitate future automation.

~/_workspace/manifests-index develop > tree -L 3  infra/templates  
infra/templates
├── application.yaml
├── deployment.yaml
├── ingress.yaml
└── service.yaml

Phew! That was just one of the 12-factors! and how we could use Git creatively for our Codebase. Stay tuned for the next 11!

Did we achieve “one codebase many deploys”? And which tools did we add to our hip belt?

If you wish to make an apple pie from scratch, you must first invent the universe.

Carl Sagan meant it!

Wrapping Up and Looking Ahead

In this blog post, we’ve journeyed through the practical implementation of (the first factor?) 12-factor microservices applications on Kubernetes. We’ve examined the intricacies of using Git as a version control system, discussed the significance of Docker and its registry, and emphasized the role of Kubernetes in the deployment of Docker images. All these components come together to create a solid foundation for the development, deployment, and management of microservices.

We’ve also shed light on the importance of a streamlined Git workflow, the judicious use of Git branches, the interplay between Docker images and Kubernetes, and the critical role of container orchestration. We’ve highlighted the leanest toolkit required to bootstrap a 12-factor microservice application, underscoring the significance of each tool in the process.

Next Steps

As next steps, I encourage you to:

1. Experiment: Apply these principles and tools to your own projects. There’s no better way to understand these concepts than by putting them to use.
2. Expand your knowledge: Dive deeper into each tool and technology we’ve discussed. Each one has much more to offer than what we’ve been able to cover in this post.
3. Stay updated: The cloud-native ecosystem is rapidly changing. Keep an eye on new developments, and don’t hesitate to explore new tools and practices that could improve your development workflow.

Remember, the ultimate goal is to build applications that are scalable, maintainable, and resilient. The tools and practices we’ve discussed here are a means to that end. Happy coding! Do reach out with questions!