Kubernetes StatefulSet Initialization with Unique Configs per Pod

How to mount a unique configuration per pod for a stateful application (e.g. how to mount separate configurations for master and slave database pods)

Photo by Ferenc Almasi on Unsplash

StatefulSets in Kubernetes are used to manage stateful application that require one or more of the following:

• Stable, unique network identifiers
• Stable, persistent storage
• Ordered, graceful deployment and scaling
• Ordered, automated rolling updates

In practice, StatefulSets are most commonly used to deploy databases (e.g. MySQL, PostgreSQL, Redis, Elasticsearch) onto Kubernetes. The declarative nature of Kubernetes makes it easy to run replicated stateful applications. For example, if you need to run multiple read-only replicas for a database, StatefulSets can be used to mount the same configurations per replica.

But what happens when you need to run different configurations per each pod in a StatefulSet cluster? A common scenarios is when you need to differentiate between a master node (read-write) and slave nodes (read-only). This can be resolved by creating a separate StatefulSet for the master and the slave nodes. What if you need a more complicated setup? Perhaps the StatefulSets run some distributed ledger network and each StatefulSet has a different role (e.g. full vs. light vs. archive nodes for Ethereum). This can be extended to a scenario where different StatefulSet pods with the same role may have different permissioning schemes or data synchronization needs depending on the network topology (e.g. pod-0 may need to synchronize all of its data with an external database vs. pod-1 can only read a subset of the data and write to a message queue).

Currently, Kubernetes does not provide a simple solution to support use cases like ones describes above. An argument can be made that each StatefulSet with such different configuration needs should be treated as separate deployments, but I wanted a way to group a cluster of StatefulSets with various roles and scale them easily.

Leveraging Ordinality

A workaround to initialize each StatefulSet pod with unique configurations leverages initContainers and pod ordinality. Kubernetes…