Solving ArgoCD Secret Management with the argocd-vault-plugin

**UPDATE: We have moved the argocd-vault-plugin repository to the argoproj-labs organizations, you can now find use here https://github.com/argoproj-labs/argocd-vault-plugin

GitOps has quickly become one of the hotter topics within the realm of DevOps. GitOps was introduced by Weaveworks in 2017 and has been trending upward ever since. I will not go into why to use GitOps in this article but you can read more about it here. One of the questions that always comes up when discussing GitOps is Secret Management. Every single talk, presentation or demo involving GitOps always has someone bringing up the question, “How do you handle secrets with GitOps?”, and that is a very good question.

As our team was evaluating moving to GitOps, we landed on using ArgoCD as our GitOps solution. ArgoCD provided a stable tool that could handle deploying hundreds of microservices across many different Kubernetes Clusters in a fast and reliable way.

Once we chose our GitOps tool, it was time to figure out what to do about the Secrets problem. At the same time, we were starting to migrate our Secrets to HashiCorp Vault, so we knew we would need something that could bridge the gap between ArgoCD and Vault. We looked around at some existing tools and one of the issues we found were that the potential solutions to this problem had a very high barrier to entry. Whether that was having to manually encrypt secrets or deploying Operators to do some of the work, none of these solutions fit well with what my team was trying to do.

So, we decided to build our own tool called argocd-vault-plugin.

What is the argocd-vault-plugin?

The argocd-vault-plugin is a custom ArgoCD plugin for retrieving secrets from HashiCorp Vault and injecting them into Kubernetes YAML files. Within ArgoCD, there is a way to integrate custom plugins if you need something outside of the supported tools that are built-in and we wanted to take advantage of this pattern. One of the ideas behind this plugin was to write it in a way that did not require an Operator or Custom Resource. This allows us to be able to parameterize any Kubernetes resources, even Custom Resources, not just Secrets.

The plugin works by first retrieving values from Vault based on a path that can be specified as an Environment Variable or an Annotation inside of the YAML file and then injects the values into a templated out yaml, that uses <> as the template markers. For example:

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  annotations:
    avp.kubernetes.io/path: "path/to/secret"
type: Opaque
stringData:
  password: <password-vault-key>

In the yaml above, you see that we have a normal Kubernetes Secret definition. However, something is not quite normal. Under data, we have a secret named password but the value is <password-vault-key> this is where the plugin will inject a value if it finds the password-vault-key key in Vault.

So when the plugin runs, it will take the avp.kubernetes.io/pathannotation from the yaml and use that to look for the secrets we want to inject into this Kubernetes Secret. So we take the avp.kubernetes.io/path annotation and combine it with the value inside the <> symbols. So the final path in this scenario would be path/to/secret with the Vault key password-vault-key. So when the plugin finishes running, we would have yaml that looks like this:

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  annotations:
    avp.kubernetes.io/path: "path/to/secret"
type: Opaque
stringData:
  password: some-password # From the key password-vault-key in Vault

You can view all the supported backends and authentication types here, https://argocd-vault-plugin.readthedocs.io/en/stable/backends/.

How to Install and use the Plugin

Prerequisites

Before starting the tutorial make sure you have these 3 things deployed/enabled. We will not show you how to do that in this article but you can find documentation below on how deploy the tools necessary.

• ArgoCD — https://argoproj.github.io/argo-cd/getting_started/
• Vault — https://www.vaultproject.io/docs/platform/k8s/helm/run
• AppRole Enabled — https://www.vaultproject.io/docs/auth/approle

Make Plugin Available to be used by ArgoCD

In order to use the plugin, we first need to download the plugin and then register the plugin with ArgoCD to use as a custom plugin. There are a couple different methods for doing this that you can read about here but for this example we are going to use the Volume Mount strategy. In order to do this, we will have to customize the argocd-repo-server deployment:

containers:
- name: argocd-repo-server
  volumeMounts:
  - name: custom-tools
    mountPath: /usr/local/bin/argocd-vault-plugin
    subPath: argocd-vault-plugin
volumes:
- name: custom-tools
  emptyDir: {}
initContainers:
- name: download-tools
  image: alpine:3.8
  command: [sh, -c]
  args:
    - wget -O argocd-vault-plugin
      https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.6.0/argocd-vault-plugin_1.6.0_linux_amd64

      chmod +x argocd-vault-plugin &&\
      mv argocd-vault-plugin /custom-tools/
  volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools

Looking back at this code, there are few things happening. First we create a blank volume named custom-tools to be used for moving the argocd-vault-plugin binary from the initContainer to the main container. Then we are downloading the argocd-vault-plugin binary via an InitContainer and then moving the binary to a path custom-tools that will be used later. We then have a Volume Mount on that custom-tools path so that we can access it in the main container. Now in the argocd-repo-server container, we are adding a Volume Mount that points to the custom-tools volume we created earlier and mounting that volume within usr/local/bin to make the plugin available to the container. So at this point, you should be able to exec or ssh into the argocd-repo-server pods and see this:

Once the plugin has been made available, the next step is to register the plugin with ArgoCD itself. This is a pretty straight forward step. There is a configMap called argocd-cm. All that is required to to go to that configMap and add:

data:
  configManagementPlugins: |-
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]

After adding the configManagementPlugins section and saving the configMap, you can then restart the argocd-repo-server deployment and then you should see the plugin as an option in ArgoCD:

Configuring the Plugin

At this point we should have ArgoCD running with the argocd-vault-plugin installed and registered to be used. The next step will be to configure the plugin to point to you instance of Vault.

There are a couple different ways to configure the plugin (Environment Variables, Kubernetes Secret, Config File) but we are going to go with using Environment Variables for the sake of this tutorial. There are 5 required variables to configure AppRole as the authentication method. Those are:

VAULT_ADDR: "http://vault:8082"     # Path to your Vault Instance
AVP_TYPE: vault                     # The Backend Type
AVP_AUTH_TYPE: approle              # Auth method we are using
AVP_ROLE_ID: role_id                # AppRole Role ID
AVP_SECRET_ID: secret_id            # AppRole Secret ID

We are going to add these into Kubernetes as a Secret within the your argocd namespace. Create a file called argocd-vault-plugin-credentials.yaml and paste this into that file. Make sure to replace the values with your actual Vault values.

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
type: Opaque
data:
  AVP_AUTH_TYPE: approle
  AVP_ROLE_ID: your_role_id
  AVP_SECRET_ID: your_secret_id
  AVP_TYPE: vault
  VAULT_ADDR: your_vault_addr

Then you want to create the secret in Kubernetes and you can do that by running this command (Make sure your Kubernetes context is pointing to the correct cluster):

kubectl apply -f argocd-vault-plugin-credentials.yaml -n argocd

The only thing left to do is to make that secret available as environment variables in the argocd-repo-server pods. Go back to the argocd-repo-server deployment where you initially added the initContainer and add an envFrom pointing to the secret you just created.

containers:
- name: argocd-repo-server
  volumeMounts:
  - name: custom-tools
    mountPath: /usr/local/bin/argocd-vault-plugin
    subPath: argocd-vault-plugin
  envFrom:
    - secretRef:
        name: argocd-vault-plugin-credentials

That is it! You should now have a configured plugin pointing to your Vault. We are now ready to test out the plugin!

Deploy ArgoCD application using the Plugin

The first thing we are going to want to do is create a secret within Vault. So go to your vault and create a secret at a path called avp/test with the key of sample and the value secret. The first thing you will want to do is enable a Vault kv-v2 store. You will also need to give your approle role id permissions to be able to access the avp path. You can find documentation at https://www.vaultproject.io/docs/concepts/policies.

vault secrets enable -version=2 avp

And then:

vault kv put avp/test sample=secret

If you then read that same path you should see something like:

====== Metadata ======
Key             Value
---             -----
version         1
====== Data ======
Key          Value
---          -----
sample       secret

Now we can move on to ArgoCD to deploy an application using the argocd-vault-plugin.

Open ArgoCD and create a new application

We are going to name it sample-secret and put it in the default project

I have a sample repo that we will use to pull a example secret file from at https://github.com/werne2j/arogcd-vault-plugin-demo

We will put the secret in-cluster (Within the cluster ArgoCD is installed) and in the default namespace

The last piece needed is to specify the argocd-vault-plugin plugin to be used

Now we can click the create button and see if it worked!

You should see an application created in the ArgoCD UI

And if you click the application, you will hopefully see this:

If so, you have successfully used the argocd-vault-plugin! We can confirm this by looking for the secret in Kubernetes and checking its value:

However we are not done yet! One of the great things about the plugin is that if the value changes in Vault we can update the value in the cluster with little effort. So update the value in vault:

vault kv put avp/test sample=new_secret

Now in ArgoCD you can do a hard refresh, this will perform a dry-run of the plugin

Now you should notice that is application is out of sync:

This means that the plugin performed the dry run and determined that the output was different than what was currently in the cluster. Now all we have to do is sync the application and we should see the application back green!

Thank you for reading! I really hope that this tool can help you along in you GitOps journey! If the argocd-vault-plugin interests you and you would like to learn more or contribute to the project you can find us on Github!

GitHub - argoproj-labs/argocd-vault-plugin: An Argo CD plugin to retrieve secrets from Secret…