Gopaddle Configurator: A Kubernetes-native Versioning System for Application Configuration

Photo by Ganapathy Kumar on Unsplash

Flamingos are filter feeders and turn their heads “upside down” to eat. When it’s time to eat, a flamingo will place its head upside down in the water with its bill pointed at its feet. It then sweeps its head side-to-side, using its tongue to pump water in and out of its bill. Comb-like plates along the edge of the bill create a filter for water to rush out, while trapping food inside. (Source: nationalzoo.si.edu)

ConfigMap and Secret API Resources

Kubernetes provides ConfigMap and Secret API resources to help decouple the Application’s configuration from the Application container. The ability to decouple the configuration from the application makes it effortless to port applications across different deployment environments and Cloud Providers. ConfigMap is used to store non-confidential data, whereas Secret is used to store confidential data like credentials and tokens. Pods can consume ConfigMaps and Secrets as environment variables, command-line arguments, or as configuration files in a volume. The usage of these resources extends beyond storing configuration data; for e.g. one can store a shell script in a ConfigMap, mount it inside an initContainer and execute the script, so that application specific init actions are performed before the application container starts up.

Versioning of ConfigMaps and Secrets

An application’s configuration data is prone to changes over the lifecycle of the application, and it’s a good practice to version control the changes to ConfigMap and Secrets. Version controlling helps in auditing, enables smooth rollback to the last working version and aids in troubleshooting application failures by allowing to inspect the previous configuration of the application. However, Kubernetes doesn’t provides an in-tree solution for versioning these resources. We’ll have to use an out-of-tree solution for this.

A versioning system for any Kubernetes API resource can be implemented in two different ways (i) by creating new instances of the API resource within the Kubernetes cluster using suitable add-ons (ii) by using external version control systems like Git. In principle, the former approach uses the etcd data store of K8s to maintain the versions, whereas the latter approach simply uses Git for versioning. In this blogpost, I’ll explain about Configurator, an open-source Kubernetes add-on, that helps in versioning and syncing ConfigMaps and Secrets.

Overview of Configurator

Configurator Logo

Configurator uses CRDs to allow users to create CustomConfigMaps and CustomSecrets, which in turn creates separate ConfigMap and Secret resources with a postfix appended to the name of the resource. When a change is detected in the CustomConfigMap or CustomSecret, Configurator automatically generates a new ConfigMap with a new postfix. This acts like a version control for the ConfigMaps and Secrets.

In order to keep the Deployments and Statefulsets in sync with the latest version of ConfigMap, users must first create a CustomConfigMap. This creates a new ConfigMap with a postfix i.e., the initial version. Users then have to reference the ConfigMap along with the postfix in their Deployment and Statefulset specifications. From them on, users can update the CustomConfigMap directly. Any change in the CustomConfigMap will be automatically rolled out to all the Deployments and Statefulsets referencing the initial ConfigMap version.

Benefits of Using Configurator

A solution like Configurator can be useful in several different ways. I’ve listed down a few of them below:

• In-cluster versioning system: A simple in-cluster versioning system for ConfigMap and Secret resources using Kubernetes itself for maintaining the versions. This can be useful in environments that do not have a Git repository, or the Cluster doesn’t have enough compute resources to run a separate Git server (e.g., Edge clusters)
• Automatic versioning: The versioning of resources is done automatically by Configurator. User just needs to update/edit the original CustomConfigMap resource.
• Automatic sync service: Configurator keeps track of the different versions of CustomConfigMap and automatically syncs the changes to the Pods that consume the ConfigMap. User doesn’t needs to track the versions and manually update and rollout the Deployments/Statefulsets.
• Immutable ConfigMaps and Secrets: Configurator helps in realizing Immutable ConfigMaps and Secrets because any modification to the resource results in creating a new version of the resource and the previous version remains unchanged.

How Configurator Works

Kubernetes allows developers to extend the kubernetes api via Custom Resources. Configurator defines custom resources of kind CustomConfigMap and CustomSecret. It implements a custom controller (named configurator-controller). Users can use kubectl commands for creation and deletion of CustomConfigMap and CustomSecret resources.

How Configurator Works

The Configurator controller uses the Informer framework to get notified when a new CustomConfigMap is created or when an existing CustomConfigMap is modified. The controller then queues the resource in the work queue. The work queue is monitored by multiple workers (implemented as Go routines). A worker’s role is to monitor the work queue and call the sync handler to process the job.

For a given CustomConfigMap, all existing versions of K8s ConfigMap are fetched using label selector. When a new version of the ConfigMap needs to be created, a new ConfigMap resource is created with the label name:<configmap name>. The latest version of the ConfigMap is identified using a special label latest:true. There’s a separate watcher routine which watches for new ConfigMaps that are managed by the Configurator. The watcher then fetches all Deployments and Statefulsets that use the previous version of the ConfigMap and then modifies the Deployment/Statefulset to use the newly created ConfigMap. This modification triggers a rolling update on the Deployment/Statefulset so the new Pods will start using the new version of the ConfigMap.

Try Configurator Now

YAML files for deploying the latest version of Configurator is available under the deploy folder for the Github repo (github.com/gopaddle-io/configurator) You need to deploy the CRDs, the controller and the RBAC configuration.

kubectl apply -f deploy/crd-customConfigMap.yaml
kubectl apply -f deploy/crd-customSecret.yaml
kubectl create ns configurator
kubectl apply -f deploy/configurator-clusterrole.yaml
kubectl apply -f deploy/configurator-clusterrolebinding.yaml
kubectl apply -f deploy/configurator-serviceaccount.yaml
kubectl apply -f deploy/configurator-deployment.yaml

Verify if the configurator controller is created successfully.

kubectl get deployment -n configurator
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
configurator-controller              1/1     1            1           4h38m

Once configurator is deployed in the cluster, start creating CustomConfigMaps. Sample CustomConfigMaps are available under artifacts/examples folder. Create CustomConfigMap. This will create a K8s ConfigMap with a postfix.

kubectl apply -f artifacts/exmaples/example-customConfigMap.yaml

List the ConfigMap and make a note of the postfix for the first time.

kubectl get configmap -n test
NAME               DATA   AGE
testconfig-srseq   1      9s

Here srseq is the postfix. Create a Deployment referencing the newly created ConfigMap. Please note the deployment metadata label must contain the config name and the config postfix that was created initially e.g. testconfig: srseq. Under the VolumeMounts section, use the complete name of the ConfigMap i.e. testconfig-srseq

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  labels:
   testconfig: srseq
    app: busybox
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
        volumeMounts:
        - mountPath: /test
          name: test-config
      volumes:
      - name: test-config
        configMap:
          name: testconfig-srseq

From now on, you can directly update the CustomConfigMap and this will create a ConfigMap with a new postfix and will automatically sync up the related Deployment with the newly created configMap. Same functionality applies for Secrets as well.

Conclusion

In this blog, I explained the importance of versioning ConfigMap and Secret resources in a Kubernetes cluster and the two different approaches to achieve this versioning. Then I delved into Configurator, an Open source add-on for implementing a Kubernetes-native in-cluster solution for versioning Application configuration. Do visit the project repository in Github (github.com/gopaddle-io/configurator) and give it a try. If you happen to know about similar solutions, do drop a note in the comments section.

👉 I tweet & blog regularly on Kubernetes and Cloud-Native Tech. Follow me on Twitter and Medium

👉 Check out kube-fledged, a kubernetes operator for creating and managing a cache of container images directly on the cluster worker nodes, so application pods start almost instantly