The Essential Guide to Upgrading From Cert-Manager v0.4
What is Cert-Manager?
Cert-manager is an open-source, Kubernetes-native controller provided by Jetstack. It can issue and manage all of the certificates in your cluster for you, and will go out and fetch LetsEncrypt certificates and renew them when they’re about to expire. Cert-manager is incredibly popular and used by many organizations to reduce the complexity of having to issue certs for encrypting internal pod-to-pod communication or for generating certs for your public endpoints that are exposed by an Ingress Controller.
I was first introduced to cert-manager when I started learning Kubernetes back in the summer of 2018. At that time we were using one of the latest versions, v0.4.0, to issue and manage our certificates within our clusters. In September of that year we looked into using v0.5.0, which introduced a breaking change to how cert-manager is installed using Helm. As version v0.4 was still fairly new, we decided to skip v0.5 and come back to it at a later date.
It’s now February 2020. A new job and role later and I’m managing Kubernetes environments again along with my old pal cert-manager. The project I’m on at the moment is still running v0.4.1, which is miles behind the latest release of v0.13.0. Why haven’t we upgraded? It’s mostly a case of the adage “if it ain’t broke, don’t fix it.” Also, you can’t leap from v0.4 to a higher version; you have to do it sequentially as there are breaking changes with each minor version (though not always).
Why Should I Upgrade?
Back in 2018, we noticed we were getting rate limited by LetsEncrypt. Since we were deploying into a sandbox environment dozens of times per day and recreating certificates for multiple domains each time (naughty, I know), we assumed we were being a bit too aggressive with our testing. It turns out this was actually an issue in earlier versions of cert-manager and has been getting fixed progressively with each release, starting with v0.6.0.
Additionally, earlier versions of the Helm chart for cert-manager include CustomResourceDefinitions. CustomResourceDefinitions, or CRDs, allow you to extend Kubernetes’ API and create resources of your own. If you create a resource definition for certificates, for example, and you bundle that into a chart, that CRD is owned and managed by Helm. If you then run helm uninstall on that chart, Helm will remove the CRD, which will then prompt the Kubernetes API to remove all objects that were associated with it. This means that in older versions of the chart your certificates were inherently bound to that installation of Cert-manager, and if you uninstall it your certificates go with it. Oddly, if you run helm upgrade and there is a change to one of the CRDs, Helm will most likely not respect the change, and the upgrade will fail. James Munnelly (from Jetstack) summed up all of the reasons for removing CRDs from the Helm chart best on this GitHub issue.
Is It Difficult to Upgrade?
The upgrade notes seem to indicate that going from v0.4.1 to v0.7.2 should be a safe jump. Anything after v0.7 requires additional groundwork, which I’ll cover later on in this post.
Part 1: From v0.4 to v0.7.2
As I mentioned before, due to Helm’s poor handling of CRDs — which cert-manager relies heavily upon to function properly — Jetstack decided to remove their CRD templates from the Helm chart entirely. Now, you must install them manually before you install the Helm chart via kubectl apply -f. Likewise, if you want to completely remove cert-manager, you will have to uninstall the Helm chart and then run kubectl delete -f to remove all of the CRDs. It’s a matter of reading the docs carefully and adding an extra step to your automation scripts.
Another point to mention is that Jetstack strongly recommends you install cert-manager into its own namespace and label it so cert-manager doesn’t attempt to validate its own self-signed certificates, which will fail. You can read more about this here.
Here are the steps we’ll walk through for upgrading from v0.4 to v0.7.2:
- Backup certificates, issuers, clusterissuers
- Remove old version of cert-manager from kube-system namespace (if you installed it there)
- Install new CustomResourceDefinitions for cert-manager via kubectl
- Create
cert-managernamespace and label it withcertmanager.k8s.io/disable-validation="true" - Install new version of cert-manager with new Helm chart into a dedicated namespace called
cert-manager - Restore backup we created earlier
Here’s what the first step (1) looks like:
Then we can remove the old version of cert-manager (2) and ensure that Helm also removed all of cert-manager’s associated CustomResourceDefinitions:
The errors above mean cert-manager is no longer present in our cluster. Before upgrading, check to make sure you don’t have old ClusterRoleBindings, ClusterRoles and ServiceAccounts floating around that are associated with a previous version of cert-manager. This could clash with the new version you are about to install. Run the following commands to ensure they are removed:
Once this is done, we can apply the new CRDs (3) for version v0.7.2 (which are no longer bundled with the chart), and then install the desired version of cert-manager (4, 5) into the cert-manager namespace:
Finally, we’ll restore the cluster issuers, issuers and certificates from our backup (6) by running `kubectl apply -f cert-manager-backup.yaml`
Now, we’ve jumped from an extremely outdated version of Helm right up to v0.7.2! To make the whole process easier for you, I’ve drafted up a script that should take care of this all in one go:
Part 2: From v0.7.2 to v0.12
Here is where I will leave you with the documentation provided by Jetstack, which outlines the steps required for each upgrade. Each version has a different requirement, some of which are optional in one that become a requirement later on. Therefore I would not recommend skipping multiple versions as you may put yourself at risk for a broken environment later on:
v0.7 to v0.8: Update ACME issuers and certificates to new format (not required but strongly recommended)
v0.8 to v0.9: Delete cert-manager deployments before applying the upgrade
v0.9 to v0.10: Delete webhook’s issuer and certificate resources (only if using static manifests)
v0.10 to v0.11: ACME issuers and certificates must be upgraded before upgrading to v0.11. Additionally there are annotation changes that reflect the new API. All of this can be found here.
v0.11 to v0.12: Removal of webhook API service
v0.12 to v0.3: No special upgrade steps required!
Further Troubleshooting
When restoring your certificates from backup, you may notice the following error: spec.dnsNames in body must be of type array: "null". While I haven’t been able to figure out why this happens, fixing it is easy. In the backup we created earlier, locate the dnsNames property in the offending certificate resource. Change it from an empty value to an empty array:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: test-ca-crt
spec:
secretName: test-ca-crt
commonName: test-ca-certificate
issuerRef:
name: ca-issuer
kind: Issuer
dnsNames: <---change this to the following below
dnsNames: []This GitHub user did an excellent breakdown of how the certs are validated against the schema, and why this particular case gives you an error.
If you want to roll back to the earlier version of cert-manager, you may also run into an issue where you cannot delete the cert-manager namespace. This can be remedied by deleting all of the CRDs you installed with the chart by running kubectl delete -f and passing the URL to the CRDs for the version of the chart you installed. Jetstack have a guide for properly removing cert-manager along with an explanation for why this occurs here.

