Terraform: don’t use kubernetes provider with your cluster resource!

Photo by Sebastian Herrmann on Unsplash

Don’t worry, you can still deploy your namespaces in the cluster, but you have to be careful.

So you use terraform to create a kubernetes cluster. This is not at all uncommon to create some resources in the cluster as well. Like some namespaces, RBAC, network policies and maybe even deploy an ingress controller. You decided that it all belongs to the infrastructure and terraform is your IaC tool.

Thankfully, terraform has kubernetes provider that can do exactly this.

Let’s take the following minimalistic example. It creates an Azure Kubernetes Service Cluster and a namespace in it:

If you apply this script, the 3 resources will be created in the correct order and you will see the green result from terraform. So far so good.

And at this point you will probably continue adding infrastructure to the project and never look back.

But as soon as you make a change in the cluster configuration that leads to cluster recreate, you will be surprised what happens when you run terraform refresh.

You will start getting strange errors.

Get Pavel Tuzov’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It can be something like this:

Error: Get “http://localhost/api/v1/namespaces/demo": dial tcp 127.0.0.1:80: connect: connection refused

Or I also saw something like this:

Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Seems like the kubernetes provider configuration is somehow reset? But why at refresh?

And no, it doesn’t help setting explicit dependency between the resources. The graph looks pretty good:

As usually, only after playing around, scratching your had, searching the internet, you will finally read carefully through the official kubernetes provider documentation:

WARNING

When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources. Please refer to this section of Terraform docs for further explanation.

Bold text is marked by me.

Hence the title of this post. You need to split cluster creation and creating resources in it into two different scripts. And preferably use terraform remote state data source to get required cluster information for the kubernetes provider.