Managing Kubernetes Secrets Dynamically from Vault via External Secrets Operator

Hello, I hope everything is going well in your life.

In this article, we will examine External Secret Operator, an Open Source software that allows us to manage Kubernetes Secrets dynamically externally.

We will install External Secret Operator on the Kubernetes Cluster via helm as a CRD (Custom Resource Definition). Then, we will create an External Secret Store and External Secret.

You can create External Secrets either on a namespace basis or globally.

The External Secret Store supports providers that allow you to dynamically transfer secrets from various platforms such as AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, HashiCorp Vault, and more, into Kubernetes secrets. In this article, we will fetch secrets from HashiCorp Vault.

We’ll quickly set up Vault in the article and then create a token with read permissions. Through this token, the External Secret Store will communicate with Vault. Finally, we’ll complete the article by creating example secrets.

1- Install Vault on Kubernetes Cluster via Helm

You can install Vault on the Kubernetes Cluster using the following command:

# Get Helm Version 
helm version
version.BuildInfo{Version:"v3.11.1", GitCommit:"293b50c65d4d56187cd4e2f390f0ada46b4c4737", GitTreeState:"clean", GoVersion:"go1.19.5"}

# Install Vault 
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update 
helm search repo hashicorp/vault
helm install vault hashicorp/vault -n demo-vault --create-namespace 

After installing Vault, it’s necessary to unseal the Vault pod from within to bring it into an unsealed state. Otherwise, you won’t be able to use Vault while it’s in the sealed state.

We’ll obtain one Unseal Key and the Root Token using the following commands.

If you want to learn more detailed information about Vault, you can check out the following series.

https://medium.com/@sametarslantrk/vault-consul

# Exec Vault Pod
kubectl exec -it -n demo-vault vault-0 -- sh 

# Run Commands Below Inside Vault Pod 
vault status
mkdir /vault/tmp &&
vault operator init -key-shares=1 -key-threshold=1 > /vault/tmp/vault-init.txt &&
cat /vault/tmp/vault-init.txt | grep "Unseal Key" | cut -d' ' -f4 > /vault/tmp/vault-unseal-key.txt &&
cat /vault/tmp/vault-init.txt | grep "Initial Root Token" | cut -d' ' -f4 > /vault/tmp/vault-root-token.txt &&
vault operator unseal "$(cat /vault/tmp/vault-unseal-key.txt)" &&
vault login "$(cat /vault/tmp/vault-root-token.txt)"
 
vault status

Following these steps, the Vault pod will fully initialize and become operational.

2- Install External Secret Store as CRD on Cluster

You can install External Secret Operator by executing the following commands.

# Add helm repo to local 
helm repo add external-secrets https://charts.external-secrets.io
helm repo update 

# Install External Secret Store 
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  # --set installCRDs=false

After executing these commands, the installation of the External Secret Operator CRD will be completed.

kubectl get crd | grep external 

3- Create Read Token on Vault

We’ve completed our basic installations up to this point. At this stage, to ensure the security of Vault, we’ll create a token with read permissions. Through this token, we’ll establish the connection between the External Secret Operator and Vault.

Firstly, we’ll create a policy with read permissions on the secret engine. Then, using this policy, we’ll complete the token creation process.

# Exec Vault Pod 
kubectl exec -it -n demo-vault pods/vault-0 -- sh

vault login <ROOT_TOKEN>

# List Policy
vault policy list 

# Write Policy Named demo-policy
vault policy write demo-policy - << EOF
path "demo/*" {
    capabilities = ["read","list"]
},  
EOF

# Read Created Policy 
vault policy read demo-policy

# Create Token has read permission and Check it 
vault token create -policy=demo-policy -ttl=200h 
vault token lookup -accessor <TOKEN_ACCESOR>

4- Create Vault Secret Engines and Secrets

After these steps, we’ll create two different Secret Engines on Vault. Then, we’ll log in to Vault using the token we created and verify whether we have access only to specific Secret Engines within the scope of the policy we provided.

# Exec Vault Pod
kubectl exec -it -n demo-vault pods/vault-0 -- sh

# Create Secret Engine 
vault secrets enable -version=2 -path="demo" kv
vault secrets enable -version=2 -path="demo-2" kv
vault secrets list

After these steps, you can verify that access is only granted to the demo secret engine by performing a port-forward operation to access the Vault interface with the read token.

kubectl port-forward -n demo-vault services/vault 8200:8200

# Exec vault pod 
kubectl exec -it -n demo-vault vault-0 -- sh
vault login <VAULLT_ROOT_TOKEN>

# List vault secret engine 
vault secrets list

# Create secrets in demo secret engine 
vault kv patch demo/secret1 secret1_key1=secret1_value1
vault kv patch demo/secret1 secret1_key2=secret1_value2
vault kv patch demo/secret2 secret2_key1=secret2_value1
vault kv patch demo/secret2 secret2_key2=secret2_value2

# List and get Secrets from demo secret engine 
vault kv list demo/
vault kv get demo/secret1
vault kv get -format=json demo/secret1

You can also confirm the created secrets from the Vault interface.

5- Create Kubernetes Secrets From Vault

At this stage, we’ll fetch our Kubernetes secrets from Vault. We’ll complete the process by creating an External Secret Store and External Secret.

5.1- Create Kubernetes Secret for Vault Read Token

A Kubernetes secret object is created for the Vault Read Token.

# secret-vault-token.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vault-read-token
  namespace: demo-app
type: Opaque
data:
  token: <BASE64_ENCODED_VAULT_READ_TOKEN>

kubectl create ns demo-app
kuebctl apply -f secret-vault-token.yaml

5.2- Create Secret Store

An External Secret Store is created to establish the connection between Vault and the Kubernetes cluster.

# secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: demo-vault-backend
  namespace: demo-app
spec:
  provider:
    vault:
      server: "http://vault.demo-vault:8200"
      path: "demo"
      # Version is the Vault KV secret engine version.
      # This can be either "v1" or "v2", defaults to "v2"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-read-token"
          key: "token"

kubectl apply -f secret-store.yaml

kubectl get secretstores.external-secrets.io -n demo-app
kubectl describe secretstores.external-secrets.io -n demo-app demo-vault-backend

At this stage, if the Secret Store has successfully established a connection with Vault, you should see Type: Ready.

5.3- Create External Secret

An External Secret is created to generate Kubernetes objects from the secrets in Vault.

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-vault-backend-secret
  namespace: demo-app
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: demo-vault-backend #Secret-store name 
    kind: SecretStore
  target:
    name: external-secret-v1 #k8s-secret-name
  data:
  - secretKey: k8s-secret1 #k8s-secret-key
    remoteRef:
      key: secret1 #vault-secret-name
      property: secret1_key1 #vault-key-value
  - secretKey: k8s-secret2 #k8s-secret-key
    remoteRef:
      key: secret2  #vault-secret-name
      property: secret2_key2 #vault-key-value

kubectl apply -f external-secret.yaml

kubectl get externalsecrets.external-secrets.io -n demo-app
kubectl describe externalsecrets.external-secrets.io -n demo-app demo-vault-backend-secret

At this stage, the External Secret should appear as Type: Ready.

Since everything is going smoothly, we can now verify the creation of our Kubernetes Secret Object.

kubectl get secrets -n demo-app
kubectl describe secrets -n demo-app external-secret-v1

kubectl get secret external-secret-v1 -n demo-app -o jsonpath='{.data.k8s-secret1}' | base64 --decode
kubectl get secret external-secret-v1 -n demo-app -o jsonpath='{.data.k8s-secret2}' | base64 --decode

Now, we will update the secret values in Vault, and then we will verify that these updated values are automatically synchronized to the Kubernetes Secrets.

# Exec vault pod 
kubectl exec -it -n demo-vault vault-0 -- sh 
vault login <VAULLT_ROOT_TOKEN>

# Update secrets 
vault kv patch demo/secret1 secret1_key1=udated_secret1_value1
vault kv patch demo/secret2 secret2_key2=updated_secret2_value2
vault kv get demo/secret1
vault kv get demo/secret2

As a result, we can see that our Kubernetes Secret automatically pulls the updated values written to Vault.

We conclude our article here.

Wishing you find it helpful.