Lessons learned from forking a Helm Project
Helm is not something that is new to me. Nor is writing my own or contributing to a Charts project. One thing that is new to me recently, however, is forking an upstream Chart project and hacking on it to add features. I learned some hard but also extremely helpful lessons that should make contributing a much easier process in the future. In this article I am going to explain what I learned this past week while working on an forked Chart project. Ill explain each lesson learned with a resolution. I did not think that I would learn some valuable lessons from such a simple task.
You must also add your fork as a Github Page.
This was something that I was not sure about as I had not yet forked a project that was using Github Pages. For context, Helm uses Github Pages to host the Chart repository. Once I had forked the repo I checked to see if my repo inherited the Github Pages settings and right away I got a “404 Not Found” error.
> helm repo add test https://geekbass.github.io/aws-efs-csi-driver/helm/Chart.yaml
W0112 16:02:38.753252 55521 loader.go:223] Config not found: admin.conf
Error: looks like "https://geekbass.github.io/aws-efs-csi-driver/helm/Chart.yaml" is not a valid chart repository or cannot be reached: failed to fetch https://geekbass.github.io/aws-efs-csi-driver/helm/Chart.yaml/index.yaml : 404 Not FoundI also confirmed from the Settings UI in my repo that it had not been published. When setting up the Github Pages for a Helm repo be sure to see the docs on the Helm site and be sure to use the specified branch “gh-pages”. Just note that your forked project will not be properly hosted by Github after simply forking a project and will require the additional steps above.
When forking you might not see all artifacts in the releases.
This was kind of a shock for me even though I have been a Github user for a long time and have forked many projects. I fully expected that if I had forked a project that it would have the exact same artifacts. In my case, the original project had the charts packaged in an archived file named helm-chart.tgz within each release. Helm is expecting to extract the file and then deploy the version based on the urls value in the index.yaml being host on the branch “gh-pages”. Because I assumed that all the artifacts, or “Assets” per Github Release Page UI, would be there when I tested the initial Deployment of the Chart, I ended up with yet another 404.
Warning DriverDeployFailure 4m36s (x2 over 7m13s) kubeaddons-controller failed to fetch https://github.com/geekbass/aws-efs-csi-driver/releases/download/v1.0.0/helm-chart.tgz : 404 Not FoundI am not sure why all of the assets were not included after I forked the repo but I was able to confirm that all of the same releases existed but without the helm-chart.tgz file (asset). Perhaps this is because it is a custom artifact that requires additional work to upload it? I didn’t look further into this because its not that important to investigate but rather note for future.
I was simply able to download the missing asset (helm-chart.tgz) from the original repo, edit the v1.0.0 release in my forked repo and upload it. Once I uploaded it, testing the deployment of the Chart worked as expected!
Pay very close attention to Releases and not just what exists on main/master branch.
After I began to get successful deployments of the Chart, I was then ready to begin adding my own features and then eventually cut my own Release of what I had built. My initial thought process and what I started doing immediately was create a new branch based on master. Once I was content with feature I would then merge that branch back to master and then cut a new release based on the changes I merged to master. This is pretty ordinary flow in my experience but what I didn’t take into consideration here was that up until now, I was testing only on the latest Chart Release version. Several other changes had been made to master since then and alot of things had been changed since the last release. Because I didn’t account for these changes, I got errors all over the place on my initial deploy. Versions for different components were off and in this case a Driver deployment was having failures due to this.
> k logs -f -n kube-system efs-csi-controller-6c65c6674f-4j6n5 csi-provisioner
I0113 18:57:49.472377 1 csi-provisioner.go:121] Version: v2.0.2
I0113 18:57:49.472472 1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0113 18:57:49.478396 1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0113 18:57:49.478854 1 common.go:111] Probing CSI driver for readiness
I0113 18:57:49.478874 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0113 18:57:49.478878 1 connection.go:183] GRPC request: {}
I0113 18:57:49.480539 1 connection.go:185] GRPC response: {}
I0113 18:57:49.480586 1 connection.go:186] GRPC error: <nil>
I0113 18:57:49.480595 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0113 18:57:49.480601 1 connection.go:183] GRPC request: {}
I0113 18:57:49.480882 1 connection.go:185] GRPC response: {"name":"efs.csi.aws.com","vendor_version":"v1.0.0-dirty"}
I0113 18:57:49.480933 1 connection.go:186] GRPC error: <nil>
I0113 18:57:49.480947 1 csi-provisioner.go:182] Detected CSI driver efs.csi.aws.com
W0113 18:57:49.480957 1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0113 18:57:49.480966 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0113 18:57:49.480976 1 connection.go:183] GRPC request: {}
I0113 18:57:49.481196 1 connection.go:185] GRPC response: {}
I0113 18:57:49.481230 1 connection.go:186] GRPC error: <nil>
I0113 18:57:49.481237 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0113 18:57:49.481242 1 connection.go:183] GRPC request: {}
I0113 18:57:49.481523 1 connection.go:185] GRPC response: {}
I0113 18:57:49.481578 1 connection.go:186] GRPC error: rpc error: code = Unimplemented desc = unknown service csi.v1.Controller
F0113 18:57:49.481597 1 csi-provisioner.go:188] Error getting CSI driver capabilities: rpc error: code = Unimplemented desc = unknown service csi.v1.ControllerAs mentioned, upon further investigation this was due to as non-compatible versions. I then realized that I should have been basing my features on the last Release that I was playing around with instead of what was currently on master. This was fine for what I was doing.
Because this is not your repo, you may not understand or know how the release process works. I will now try to identify the differences between the latest Release and master. Also, if possible, I will try to understand the release process a bit better to avoid unexpected issues.
Check for Github Actions Directory.
This goes for any repo that is forked and not just from this experience. Always be aware of a .github/ directory in the root of your project as this is used for Github Actions. I did not have any issues or unexpected things happen when I was committing or merging except for the fact that I would get an annoying email every time stating that my repo has “failure”. The point here is that not paying attention to this could potentially result in unexpected things happening that you were originally unaware of. Just wanted to point that out here. In my particular case this was the workflow used to archive the helm-chart.tgz file and cut the new release.
The Range Flow Control in a Helm chart.
The main reason for forking this project in the first place was because I wanted to add a feature that would allow for the Charts to deploy one or more PersistentVolumes (PVs), PersistentVolumeClaims (PVCs) and/or a StorageClass. After reviewing the specs for all of the Kubernetes resources, I decided that I would need a way for creating multiple instances of the same resource while using only a single template file. (IE: When a user wants to create two PVs it will create both of these using the same persistent-volume.yaml template file for creation.) This would be best served through a Loop condition of key value pairs (a map in this case) but was unsure how to do this within Helm. First Google search revealed the Range flow control structure.
From here I decided how I would like these values to be assigned within a values.yaml file and I came up with something like below (a bit stripped down):
peristentVolumes:
pv1:
annotations:
provisioned-by: efs-csi-driver
storageCapacity: 5Gi
storageClass: efs-storageclass
fileSystemId: fs-blahblah
pv2:
storageCapacity: 10Gi
storageClass: efs-sc
fileSystemId: fs-woohooThe idea above is to create two PVs named “pv1” and “pv2” and assign the underneath attributes according to its map. We need to “Loop” through each map (pv1, pv2) and then each key value pair to get what we want. We will set this as a Helm Variable of {{- range $k, $v := .Values.peristentVolumes }}. (See Go Template Variable for more information) The final solution that I came up with looks like below ./templates/pv.yaml:
{{- range $k, $v := .Values.peristentVolumes }}
apiVersion: v1
kind: PersistentVolume
metadata:
{{- if $v.annotations }}
annotations: {{ toYaml $v.annotations | nindent 4 }}
{{- end }}
name: {{ $k }}
spec:
capacity:
storage: {{ $v.storageCapacity }}
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: {{ $v.storageClass }}
csi:
driver: efs.csi.aws.com
volumeHandle: {{ $v.fileSystemId }}
---
{{- end }}To test this locally to see if our template works as expected:
> helm template ./
W0117 09:14:26.723511 43895 loader.go:223] Config not found: admin.conf
---
# Source: foochart/templates/pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
provisioned-by: efs-csi-driver
name: pv1
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-storageclass
csi:
driver: efs.csi.aws.com
volumeHandle: fs-blahblah
---
# Source: foochart/templates/pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv2
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-woohooOne additional embarrassing thing to note here. In my case, because I wanted to create multiple of the same resource from a single template file, I forgot to put--- to separate the resources. The K8s API does not like this format.
Nothing like throwing yourself into something that should be so simple and end up banging your head off the keyboard and questioning your career choices. I mean this honestly (for the most part). With every struggle comes increased knowledge no matter how small. This keeps me grounded as an Engineer. I never expected to learn some simple lessons from this little journey but hopefully there is something in this article that may help someone else out there banging their head off their keyboard.
