Secrets injection at runtime from external Vault into Kubernetes — POC

When you work in a multi cloud environment, you can't always use AWS secrets manager for storing all your secrets. Hashicorp Vault is an awesome solution for storing and managing your secrets. In this POC I will show how you can easily and directly inject secrets into your pods thanks to https://github.com/hashicorp/vault-k8s. A lot of tutorials demonstrate how to use vault when it is running IN the cluster, but here we'll use an external running vault.

Understand the dataflow

Before diving into the code, the most important thing is to understand how the vault integration is done with Kubernetes, what components do and how the all play together.

Kubernetes side

The flow starts within a pod where you want to inject your secrets. This pod is running with a Service Account (ex: sa-awesome-app). This SA logs into theVault server and asks to assume a role (awesome-app). If the role is allowed to read the secret, it is returned to k8s and injected as a file in a volume mounted in the pod.

Vault side

Initially, Vault receives the JWT from sa-awesome-app. To verify if this JWT is correct, Vault asks k8s to validate the token. Vault is configured to bind a k8s service account with a role. This role has a policy which grants it permission to read the secret path (secret/my-awesome-app/config).

For k8s to be able to validate the token, we need another Service Account (sa-vault-auth) of type ClusterRole to delegate the authentification.

Hands-on

Now that we have a better view of how vault is integrated into kubernetes, let's build our component. This POC is available on github: https://github.com/sylwit/vault_injection_k8s_poc.

If you don't want to wait any longer, just clone the repo, open the Makefile and run each target in order.

Prerequesites

• A kubernetes cluster. Here I will use minikube with docker driver.
• Kubectl
• Helm
• Docker (do I really need to specify)
• Docker-compose

Prepare our k8s environment

TL;DR : make k8s

Start our cluster, create a namespace "demo" to deploy our awesome app, set it as the default namespace.

kubectl create namespace demo
kubectl config set-context --current --namespace=demo

Create Service Account sa-vault-auth with its secret and bind it to the ClusterRole system:auth-delegator. This SA will be used by vault to validate JWT token.

kubectl apply -f k8s.yaml

Install hashicorp vault via the official helm chart. We need to set injector.externalVaultAddr to the vault server and this URL must be accessible from your cluster. I'm fetching the IP in the makefile

GATEWAY_IP=$(shell minikube ssh "grep host.minikube.internal /etc/hosts" | awk '{print $$1}')
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --set "injector.externalVaultAddr=http://${GATEWAY_IP}:8200"

Prepare our vault

TL;DR : make vault

Vault will be started with docker-compose and needs a .env file. (make .env)

echo "TOKEN_REVIEW_JWT=`kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode)}`" > .env
echo "KUBE_CA_CERT_B64=`kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'`" >> .env
echo "KUBE_HOST=https://`minikube ip`:8443" >> .env
docker-compose run -d vault

If you want to access Vault UI, navigate to http://localhost:8200 and the token is : root. It is hard-coded in the docker-compose, otherwise you can read it from docker-compose logs vault

Once vault is running, we create a secret then enable and configure kubernetes authentication.

vault login root
vault kv put secret/my-awesone-app/config username='my_user' password='my_pwd'
vault auth enable kubernetes
echo $KUBE_CA_CERT_B64 > .cert.pem
base64 -d .cert.pem > cert.pem
rm .cert.pem

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert=@cert.pem

We now write the policy to allow read permission to this secret and bind it to the SA sa-awesome-app.

vault policy write my-awesome-app - <<EOF
path "secret/data/my-awesome-app/config" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/awesome-app \
        bound_service_account_names=sa-awesome-app \
        bound_service_account_namespaces=demo \
        policies=my-awesome-app \
        ttl=24h

I wrap all this set up in a vault-init.sh script and execute it on the running vault container

VAULT_INIT=`cat ./vault-init.sh`
docker-compose exec vault sh -c "${VAULT_INIT}"

Deploy our app

TL;DR : make app

The application is a simple HTTP server which reads a file and serves its content. The repo is available here: https://github.com/sylwit/http_filereader

We also expose our awesome app through a service running on port 30100.

The full code is available in this file : https://github.com/sylwit/vault_injection_k8s_poc/blob/main/app.yaml

The magic happens in these annotations

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "awesome-app"
  vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/my-awesome-app/config"
  vault.hashicorp.com/agent-inject-template-credentials.txt: |
    {{- with secret "secret/my-awesome-app/config" -}}
    username: {{ .Data.data.username }}
    password: {{ .Data.data.password }}
    {{ end -}}

• role defines the role we assume in vault
• agent-inject-secret-credentials.txt defines the filename where the content of “secret/my-awesome-app/config” will be write to. vault-k8s will mount a volume at /vault/secrets so the final file will be accessible at /vault/secrets/credentials.txt
• agent-inject-template-credentials.txt formats the content of the file
• We set FILEREADER_CHROOT env variable to /vault/secrets as our base folder

kubectl apply -f app.yaml

Wait few minutes then navigate to http://`minikube ip`:30100?file=credentials.txt and voila.

Init and side containers

k8s-vault integration can be done thanks to init and side containers.

Before the application starts, the init container will get the secret and inject it in the volume located as /vault/secrets as mentioned before.

The volume is mounted in the app container so immediately accessible.

A side container is also started, every 5 minutes, it checks if the secret has changed and updates the volume for us if needed.

Conclusion

I really like the fact that the side container avoids a new release when the secrets change. It looks a bit tedious at the beginning to configure all the resources but once it's done and scripted it's pretty easy and reusable.