Helm + Kustomize + Raw Manifests Combination

Streamlining Kubernetes Manifest Management: Combining Helm, Kustomize, and raw manifests to build the final manifests

Photo by Vedrana Filipović on Unsplash

Introduction

Navigating the Kubernetes ecosystem often entails juggling multiple tools and techniques to achieve desired deployment outcomes. Enter Helm, Helmfile, Kustomize, and Raw Manifests, each offering distinct advantages in managing Kubernetes manifests. In this article, we explore the synergies between these tools, presenting an innovative algorithm that combines their strengths to create highly customizable manifests. By seamlessly integrating Helm, Kustomize, and Raw Manifests, this approach ensures adaptability and efficiency in Kubernetes deployment workflows. Join us as we delve into the intricacies of this integration, offering insights into its implementation and potential benefits for Kubernetes practitioners.

Vocabolary

• Helm: Helm is a popular tool for creating Kubernetes manifest packages and sharing them. Homepage: https://github.com/helm/helm
• Helmfile: it’s a tool for managing different Helm packages in one specification file. For example, you may need to deploy your application with a Redis and a PostgreSQL server, so you would need to apply three Helm charts. However, using Helmfile, you can manage them more easily with just one specification file. Homepage: https://github.com/helmfile/helmfile
• Kustomize: it’s another tool for creating Kubernetes manifest packages with a structure that differs from Helm. This model creates a base manifest and some patches for each of your environments. Homepage: https://github.com/kubernetes-sigs/kustomize
• Raw Manifests: This is a basic type of manifest writing without any markup, which you can provide to kubectl to apply to Kubernetes.

Why Combine Tools?

you might need to change some default templates of a Helm chart. In such cases, you’ll need to create a customized Helm chart and store it in an artifact registry. Using a customized Helm chart can cause a divergence from the original Helm chart and prevent you from receiving updates. Additionally, there may be times when you need to include some raw manifests with your Helm chart, but you cannot add them to the original Helm chart because it is not maintained by you. You can only download it and are unable to update it. What should you do in this situation? In this article, I will describe an algorithm that combines Helm charts, Kustomize, and raw manifest files. This approach allows you to take advantage of all these tools by merging them into a highly customizable set of manifests.

Flow Chart

In the following flow chart, you can see that we are using various tools such as Helmfile and Kustomize, along with some raw manifest files, to build the final manifests and apply them to Kubernetes.

In the above diagram, you can see that I started first with Helmfile, then Kustomize, and lastly with the raw manifests. As you may know, with this order, the raw manifests have the highest priority and can override other manifests generated by Kustomize or Helmfile. You can change the order based on your requirements.

Directory Structure

In the following section, you can see that I have placed different types of files beside each other:

.
├── helmfile.yaml
├── kustomization.yaml
├── raw-manifests
│   └── networkpolicy.yaml
├── patch.yaml
├── .gitlab-ci.yml
├── .git
└── values.yaml

In the above Git repository, you can see 3 different types of files:

1- Helm assets

The helmfile.yaml and values.yaml files are related to downloading and building the Helm chart and placing them into the target directory. Here’s a sample helmfile.yaml file:

repositories:
  - name: registry
    url: https://cloudnativeapp.github.io/charts/curated

releases:
  - name: rabbitmq-proxy
    namespace: test
    createNamespace: false
    chart: registry/envoy
    version: 1.5.0
    values:
      - ./values.yaml

As you can see in the above file, it will download a Helm chart from GitHub and provide the values.yaml file to override the default values.yaml file.

2- Kustomize assets

The kustomization.yaml and patch.yaml files are used to apply some changes to the generated manifest from Helm. Here is the sample file:

---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - resources.yaml
patchesStrategicMerge:
  - patch.yaml

---
# patch.yaml
$patch: delete
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: envoy-proxy-rabbitmq

As you can see in the above file, it will delete a PodDisruptionBudget resource, which is not possible to delete through the values.yaml file of the Helm chart because the author of the Helm chart did not provide any ability to disable it. However, we can delete it by using Kustomize.

3- Raw Manifest assets

The raw-manifests/networkpolicy.yaml file is related to Raw Manifests, which need to be applied alongside other generated manifests.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
    env: beta
  name: to-dev-beta
  namespace: test
spec:
  egress:
  - ports:
    - port: 5672
      protocol: TCP
    to:
    - ipBlock:
        cidr: 192.168.1.1/32
    - ipBlock:
        cidr: 192.168.1.2/32
  podSelector:
    matchLabels:
      env: beta
      networkpolicy/to-dev: allow
  policyTypes:
  - Egress

It’s not possible to add the above file to the original Helm chart because we don’t have access to modify it. However, we need to include it in our final manifests, so it must be placed as a raw manifest in the raw-manifests directory.

Pipeline Code

The following code corresponds to the above flow chart. I used GitLab as a pipeline executor in this code, but you can adapt it to your preferred pipeline executor. The .gitlab-ci.yml file can be structured like this:

stages:
  - deploy

deploy:
  stage: deploy
  image: docker.io/mlkmhd/cd-pipeline:1
  script: |
    set -xe
    namespace="test"

    if [ -f "helmfile.yaml" ]
    then
      helmfile template > resources.yaml
    fi

    if [ -f "kustomization.yaml" ]
    then
      kustomize build . > resources.yaml
    fi
    
    kubectl -n ${namespace} apply -f resources.yaml

    if [ -d "raw-manifests" ]; then
      kubectl -n ${namespace} apply -f raw-manifests
    fi

You can read my other article, which is about using GitLab to manage different Kubernetes clusters as code:

Gitlab for Enterprise Continues Delivery

You can find all the code used in the above article at the following GitHub repository:

devops-diagrams/helmfile-and-kustomize/assets at main · mlkmhd/devops-diagrams

Conclusion

In the dynamic landscape of Kubernetes deployment management, the integration of Helm, Helmfile, Kustomize, and Raw Manifests emerges as a powerful solution for orchestrating highly customizable manifests. By leveraging the strengths of each tool, practitioners gain unprecedented flexibility and control over their deployment workflows. From seamless management of Helm packages to fine-grained configuration with Kustomize and inclusion of raw manifests, this approach empowers teams to adapt to diverse deployment scenarios with ease. As Kubernetes environments continue to evolve, this integrated approach stands as a testament to the adaptability and innovation driving the Kubernetes ecosystem forward.

Feedback

If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.