Add topology labels to your Kubernetes Pods

I had a customer ask me today: How can I add Availability Zone and Region labels to my Pods?

Photo by Joey Huang on Unsplash

In most managed Kubernetes services, like Amazon EKS, Kubernetes nodes have a label topology.kubernetes.io/zone. This label contains the name of the Availability Zone (AZ) in which the node is located. One possible way of adding an AZ label to Pods would be to get the topology label from the associated node.

Kubernetes doesn’t add topology labels to Pods currently, so I started exploring if there’s a controller that could add the labels. We already had Kyverno installed in the cluster, so if we could use Kyverno, it would save everyone a lot of time. Luckily, the solution was easier than we thought.

Kyverno is a policy engine designed for Kubernetes. It simplifies writing Kubernetes policies. Its distinct advantage lies in its user-friendly approach. Unlike other Kubernetes policy like OPA, you don’t have to learn Rego or another language to author policies.

Kyverno Mutate Rules

Kyverno finds its primary utility in performing typical operations such as mutation or validation upon the creation of resources like Pods, Services, or Deployments. For example, this policy lets you automatically add a label when you create a Pod.

The problem is that the AZ in which the Pod will land is unknown until the Pod gets scheduled and a node is assigned. To address this, we needed a mutating rule that adds label Pods after they get scheduled.

We solved this issue by using Kyverno Background Controller with Pod/binding as a resource for Kyverno to act upon, instead of the Pod itself. Then, we could use Kyverno’s mutateExistingOnPolicyUpdate attribute, which enabled us to update an existing resource.

Let’s see this in action.

Configure Kyverno

There are two things you need to make this policy work:

1. Configure Kyverno to monitor Pod/binding resource
2. Give Kyverno backgroundController the permissions to update Pods

If you’ve installed Kyverno with default configuration, chances are your installation excludes Pod/binding resource type. You can check if this resource type exists in the resourceFilter list in kyverno configMap in your cluster.

Here’s a screenshot of kyverno configMap in my cluster after installing Kyverno using the official Helm chart:

Kyverno default ConfigMap

Notice that [Pod/binding, *.*] is included in the resourceFilters. We need to remove the Pod/binding resource from the list.

You can edit the configMap to remove or generate a values.yaml to use Helm to reconfigure Kyverno. The command below generates a values.yaml:

curl "https://raw.githubusercontent.com/kyverno/kyverno/main/charts/kyverno/values.yaml" | sed  's/Pod\/binding//g' > values.yaml

Update Kyverno backgroundController:core role

Next, we need to give the Kyverno the permission to update Pods. Kyverno Background Controller uses kyverno-background-controller service account. This service account is bound to kyverno:background-controller ClusterRole.

By default, this role doesn’t have the permission to update or patch Pods. We have to patch the role to add this permission. You can add the permission in-place by using:

kubectl \
  --namespace kyverno \
  edit clusterrole kyverno:background-controller:core

I patched kyverno:background-controller:core ClusterRole to add permissions. Be careful as running the command below may override any custom permissions you've added over Kyverno's default installation:

kubectl patch clusterrole kyverno:background-controller:core \
  --type='json' -p='[{"op": "add", "path": "/rules/6/resources", "value":["pod", "configmaps", "secrets", "resourcequotas", "limitranges"]}]'

This is how the ClusterRole looks like:

Permit Kyverno to update or patch Pods

With the permissions setup, the last thing you need is the Kyverno policy that copies topology labels from nodes and adds them to Pods.

Kyverno Policy

This policy below copies the topology.kubernetes.io/zone and topology.kubernetes.io/region labels from the node to the Pod. Create a policy manifest:

cat <<EOF > add-topology-labels-to-pods.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/title: Add scheduled Node's topology labels to a Pod
    policies.kyverno.io/description: >-
      Containers running in Pods may sometimes need topology labels. This policy watches for then mutates
      the /binding subresource of a Pod to add topology.kubernetes.io/zone and topology.kubernetes.io/region labels on Pods. 
  name: add-topology-labels-to-pod
spec:
  admission: true
  background: false
  failurePolicy: Fail
  mutateExistingOnPolicyUpdate: true
  rules:
  - context:
    - name: node
      variable:
        default: ""
        jmesPath: request.object.target.name
    - apiCall:
        jmesPath: metadata.labels."topology.kubernetes.io/zone" || 'empty'
        method: GET
        urlPath: /api/v1/nodes/{{node}}
      name: ZoneLabel
    - apiCall:
        jmesPath: metadata.labels."topology.kubernetes.io/region" || 'empty'
        method: GET
        urlPath: /api/v1/nodes/{{node}}
      name: RegionLabel
    match:
      any:
      - resources:
          kinds:
          - Pod/binding
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            topology.kubernetes.io/zone: '{{ ZoneLabel }}'
            topology.kubernetes.io/region: '{{ RegionLabel }}'
      targets:
      - apiVersion: v1
        kind: Pod
        name: '{{ request.object.metadata.name }}'
        namespace: '{{ request.object.metadata.namespace }}'
    name: add-topology-labels-to-pod
    skipBackgroundRequests: true
  validationFailureAction: Audit
EOF

Apply the policy:

kubectl apply -f add-topology-labels-to-pods.yaml

This policy first gets the region and zone labels from the associated node using Kubernetes API. Then it adds the labels to Pods. Please see Kyverno documentation to learn more about getting data from Kubernetes API Server calls.

Validate labels

Create a new Pod and you should see the new labels.

kubectl run test --image=nginx
kubectl get pods test --show-labels

You should see topology labels in the output:

Topology labels in a Pod

Conclusion

Adding Availability Zone and Region labels to Pods can be a crucial requirement in many scenarios. You can use labels to limit the Pods that Prometheus scrapes metrics from, which can result in cost savings.

By utilizing Kyverno’s mutate rules and background controller, we could dynamically add topology labels to Pods after they were scheduled. This approach streamlined the process, eliminating the need to rely on a complex solution.

Reference: https://github.com/kyverno/policies/issues/887