Effective Secrets with Vault and Kubernetes

Introduction

Kubernetes is the de-facto standard for container orchestration, and Vault by HashiCorp is the de-facto standard for secrets management. Now the question is: how do you combine those technologies to use secrets from your central Vault instance in your Kubernetes applications?

One solution would be to use the AppRole auth method. Boostport provides an excellent integration of AppRoles in Kubernetes. Another possibility is to use the Kubernetes auth method. This auth method establishes a trust relationship between Vault and your Kubernetes cluster so you can use a service account to authenticate to Vault. You can use the Vault agent with Kubernetes to get and renew authentication tokens.

In this walk-through article, I will show how you can achieve the same thing with some Go helper tools that authenticate and renew tokens and even go a step further—synchronize a predefined subset of secrets from Vault to Kubernetes.

Level: advanced

Prerequisites

For the sake of simplicity, I made some choices:

• A Kubernetes cluster can be set up in many different ways. Often, minikube is used for test or development purposes. I will use kubeadm because it is so simple to set up a real cluster.
• In Kubernetes, the default namespace will be used.
• Vault will be run in development mode. Don’t use it like this in production! Make sure to set the environment variable VAULT_ADDR accordingly.
• Ubuntu will be used for all the code samples. They have been tested on a single Ubuntu 18.10 VM on GCE with 2 vCPUs and 7.5 GB. (Check out the GCP $ 300 free tier, just saying…)
• Bash will be used unless otherwise noted.

Kubernetes

Let’s get started with a simple test cluster. Below you will find the setup instructions for a single node installation.

Vault

Installation

Vault installation is pretty straightforward: download and unpack the binary:

Running a Vault Server

We will run a Vault server in development mode. Again, this is pretty easy. Please note that when starting a dev server, a root token will be written to $HOME/.vault-token, even for the root user. The Vault process will be put in the background with the & sign so that we can continue to use the same shell.

Configuring the Kubernetes Auth Method

Now we have to make sure that Kubernetes is able to talk to Vault by enabling the Kubernetes auth method. This establishes a trust relationship between Kubernetes and Vault. The named role vault-demo-role will map the policies and define a TTL.

Because we set up our Kubernetes cluster using kubeadm it’s pretty easy to locate the certificate authority (CA) store for the property kubernetes_ca_cert. This might be a bit more difficult when using a cloud-provided Kubernetes installation.

Role-based access control (RBAC)

On the Kubernetes side, we now have to set up the corresponding RBAC stuff. First, we will create a service account named vault-serviceaccount. We will then add a cluster role binding named vault-closterrolebinding so that our newly created service account is allowed to do delegated authentication requests using the default cluster role system:auth-delegator. The role vault-secretadmin-role and the role binding vault-secreatadmin-rolebinding are bound to vault-serviceaccount as well so that we are able to synchronize secrets.

Let’s apply these manifests:

Preparation is finished. Now we’re good to go with our use cases.

Use Cases

We will cover three use cases:

• The first example will demo how to authenticate to Vault and obtain an authentication token by using an init container.
• The second example will demo how this token can be renewed using a sidecar container.
• The third example will demo how to synchronize secrets from Vault to Kubernetes.

All three use cases are based on three Docker images built by my colleagues at PostFinance. Special Kudos go to Marc Sauter who wrote the initial implementation inspired by the works of Seth Vargo. All three images — made available on Docker Hub — contain little Go helper tools, the source can be found on GitHub.

Authentication with an Init Container

The first example will show the usage of the vault-kubernetes-authenticator image (auther for short). The auther runs in an init container, authenticates to Vault using the service account vault-serviceaccount , and writes the Vault authentication token to /home/vault/.vault-token.

Let’s apply this manifest and then we will do some tests to verify everything works ok.

Token Renewal with a Sidecar

The second example will show the usage of the vault-kubernetes-token-renewer image (renewer for short). The renewer runs in a sidecar container, checks the TTL periodically, and renews the authentication token accordingly.

Let’s apply this manifest as well and again let’s do some verification. (I deleted the previous deployment.)

Synchronizing Secrets from Vault to Kubernetes

The third example will show the usage of the vault-kubernetes-synchronizer (syncer for short). The syncer can be used in different ways. In the demo, a Kubernetes job will be used to do a one-off synchronization of Vault secrets from predefined paths. The Vault secrets will be written to corresponding Kubernetes secrets.

Again, let’s apply this manifest and let’s check that everything is working as expected:

The syncer could also be used in a Kubernetes cron job to periodically sync the Vault secrets or in another init container in a Kubernetes deployment so that secrets are always up-to-date.

Please note that Kubernetes secrets are not protected very well. By default, they are only base64 encoded and stored just like that in etcd as pointed out in Seth Vargo’s recent FOSDEM talk. You should enable encryption of secret data at rest. Please also make sure that you only sync those secrets which are effectively used by your Kubernetes applications, protected by corresponding Vault policies and named roles. Apart from that this approach lets you use secrets in a cloud-native manner. Your applications don’t have to access Vault directly and secrets can be injected as environment variables.

Conclusion

Both technologies, Kubernetes and Vault, can be used in a best-of-breed manner by combining and integrating them. The integration is non-trivial but is still feasible. In this article, I’ve shown you how to achieve this integration and I hope it will be helpful to you, too.

You may ask yourself why you should be using some third-party images given the fact that the official Vault image can be used to run an agent to achieve the same thing. The reasons are: the Vault agent needs a config file instead of environment variables which means you have to manage another config map. And the agent is currently not able to synchronize secrets. Additionally, the images are more lightweight. The official Vault image is around 100 MB in size. The auther and renewer images are around 10 MB and the syncer is around 40 MB.