Vault cluster with auto unseal on Kubernetes
Here again with another interesting Vault topic.
This is a script to implement a simple Vault cluster which is integrated into the Consul cluster as back storage, and a Vault as a transit server to automatically unseal vault cluster, of course for test purpose.
Before explaining the script let’s review some introduction.
What is Vault?
Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.
What is Consul?
Consul uses service identities and traditional networking practices to help organizations securely connect applications running in any environment.
What is Transit server?
Vault supports opt-in automatic unsealing via transit secrets engine. This feature enables operators to delegate the unsealing process to a trusted Vault environment to ease operations. The Transit seal configures Vault to use Vault’s Transit Secret Engine as the auto unseal mechanism.
FIND CODES HERE:
GET STARTED:
As you can find here, Transit secrets engine in vault is able to store keys for your vault cluster in order to unseal it automatically.
For this setup, we need:
1- vault cluster with consul client for each vault node
2- another vault instance as a transit secret engine
3- consul cluster to store data of our nodes
We have a script to implement and configure all of them on Kubernetes.
Let’s understand the script “vault-cluster.sh”:
1. we clean the environment by removing the “vault-cluster” namespace and its persistent volumes. Then start with deploying Consul cluster.
kubectl create namespace vault-cluster
kubectl apply -f consul/cm.yaml
kubectl apply -f consul/svc.yaml
kubectl apply -f consul/deploy.yaml REPLICAS=$(kubectl get statefulsets.apps --namespace=vault-cluster -o custom-columns=:.spec.replicas consul|tail -1)
REPLICA=$(expr ${REPLICAS} - 1)while [[ ! $(kubectl get po --namespace=vault-cluster --field-selector status.phase=Running|grep consul-$REPLICA ) ]]
do
echo -ne .
sleep 5s
done
As you are able to find files on my GitHub :
cm refer to ConfigMaps.
svc refer to Service.
deploy refer to the main file for deployment.
2. We deploy Vault transit server, and Vault cluster.
3. Let’s configure Vault transit server:
# FIRSTLY initialize the vault
kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault operator init -format=yaml > vault-auto-unseal-keys.txt # SECONDLY unseal it
kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-auto-unseal-keys.txt |head -2|tail -1|sed 's/- //g')kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-auto-unseal-keys.txt |head -3|tail -1|sed 's/- //g')kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-auto-unseal-keys.txt |head -4|tail -1|sed 's/- //g')
It’s time to enable secrets, policy and token:
VAULT_AUTO_UNSEAL_ROOT_TOKEN=$(grep root_token vault-auto-unseal-keys.txt |awk -F: '{print $2}'|sed 's/ //g')kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault login -tls-skip-verify ${VAULT_AUTO_UNSEAL_ROOT_TOKEN} kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault audit enable file file_path=/vault/logs/audit.log kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault secrets enable transit kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault write -f transit/keys/autounseal kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault policy write autounseal /vault/unseal/autounseal.hcl kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- vault token create -policy="autounseal" -wrap-ttl=12000 -format=yaml > .vault-auto-unseal-token.txt
Vault wrapping:
In many Vault deployments, clients can access Vault directly and consume returned secrets. In other situations, it may make sense to or be desired to separate privileges such that one trusted entity is responsible for interacting with most of the Vault API and passing secrets to the end consumer.
When a response is wrapped, the normal API response from Vault does not contain the original secret, but rather contains a set of information related to the response-wrapping token: TTL, Token, Creation_time, Creation Path, Wrapped Accessor.
Finally: Unwrap the token, returning the response inside. The response that is returned will be the original wire-format response; it can be used directly with API clients.
Let’s make it automatic:
VAULT_AUTO_UNSEAL_TOKEN=$(grep token: .vault-auto-unseal-token.txt|awk '{print $2}'|tr -d '\r')kubectl exec --namespace=vault-cluster --stdin --tty $TRANSIT_SERVER_NAME -- env VAULT_TOKEN=${VAULT_AUTO_UNSEAL_TOKEN} vault unwrap -format=yaml > .vault-unwrap-token.txtVAULT_AUTO_UNSEAL_TOKEN=$(grep client_token: .vault-unwrap-token.txt|awk '{print $2}'|tr -d '\r')
4. Next step is configure vault cluster and consul client ConfigMaps:
tee vault/cm-updated.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-config
namespace: vault-cluster
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/creator: hossein-yousefi
app.kubernetes.io/stack: vault-cluster
data:
config.hcl: |
listener "tcp" {
tls_disable = 1
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
}
storage "consul" {
address = "consul:8500"
path = "vault/"
}
disable_mlock = true
seal "transit" {
address = "http://vault-auto-unseal:8200"
token = "${VAULT_AUTO_UNSEAL_TOKEN}"
disable_renewal = "false"
key_name = "autounseal"
mount_path = "transit/"
tls_skip_verify = "true"
}---apiVersion: v1
kind: ConfigMap
metadata:
namespace: vault-cluster
name: consul-client-config
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/creator: hossein-yousefi
app.kubernetes.io/stack: vault-cluster
data:
consul.json: |
{
"server": false,
"datacenter": "dc1",
"data_dir": "/consul/data/",
"bind_addr": "0.0.0.0",
"client_addr": "0.0.0.0",
"retry_join": ["vault"],
"log_level": "DEBUG",
"acl_enforce_version_8": false
}
EOFkubectl apply -f vault/cm-updated.yaml &> /dev/nullkubectl rollout restart statefulsets --namespace=vault-cluster vault
5. Last step is just initializing vault-0:
kubectl exec — namespace=vault-cluster — stdin — tty vault-0 -c vault — vault operator init -format=yaml > vault-keys.txtkubectl exec — namespace=vault-cluster — stdin — tty vault-1 -c vault — vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-keys.txt |head -2|tail -1|sed ‘s/- //g’)kubectl exec — namespace=vault-cluster — stdin — tty vault-1 -c vault — vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-keys.txt |head -3|tail -1|sed ‘s/- //g’)kubectl exec — namespace=vault-cluster — stdin — tty vault-1 -c vault — vault operator unseal -tls-skip-verify $(grep -A 5 unseal_keys_b64 vault-keys.txt |head -4|tail -1|sed ‘s/- //g’)
Finally: We have a Vault cluster which can unseal itself using another vault server as transit secret engine.
Find all files on my GitHub, and do not hesitate to ask your questions.
