Hardening Monitoring: a step-by-step guide

Published in

ITNEXT

5 min readNov 16, 2022

Internal Kubernetes services are already on a private network if not exposed to the outside world. But Is there any need for hardening internal services? Before even Kubernetes became mainstream, sensitive data at data centers were applied to layers of perimeter security only when data were traveling in and out of data centers. However, things will go wrong when the internal networks get compromised. In Kubernetes, imagine a vulnerable pod becomes a target and eventually compromised. This can pose a threat to all unencrypted traffic inside a cluster. Certificates are primarily used for encryption which can mitigate this. However, without proper authentication, these pods can be used to obtain sensitive information from other pods in the network. To this end, certificates also provide infrastructure for identification and authentication inside internal networks.

In this article, I walk through how to serve metrics-server, prometheus-server and prometheus-adapter securely.

The Boring Part

source: giphy.com

We need to generate three certificates for each of the components.

Kubernetes provides API for provisioning TLS certificates that are signed by a built-in Kubernetes signer or custom certificate authority.

To request a certificate, we use the Cloudflare command line tool cfssl similar to the official guide.

Installing is by getting it through

$ go get github.com/cloudflare/cfssl/cmd/cfssl

or by using Homebrew

$ brew install cfssl

Generating private key

The next step is to generate a private key for each of our services by defining certificate signing request in a JSON file:

{
  "hosts": [
    "<service name>.<namespace>",
    "<service name>.<namespace>.svc.cluster.local",
    "<service name>.<namespace>.pod.cluster.local",
    "<service IP>",
    "<Pod IP>"
  ],
  "CN": "<service name>.<namespace>.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

For example, for metrics-server inside kube-system namespace the file will look like:

// csr-metrics-server.json
{
  "hosts": [
    "metrics-server.kube-system",
    "metrics-server.kube-system.svc.cluster.local",
    "metrics-server.kube-system.pod.cluster.local",
    "<service IP>",
    "<Pod IP>"
  ],
  "CN": "<service name>.<namespace>.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

Then apply the file using the following command:

$ cfssl genkey <csr filename>.json | cfssljson -bare server

Create Signing Request

Now, it has generated server.csr, server-key.pem. The former contains certification request, and the latter contains the encoded private key.

Next, let’s leverage Kubernetes API to create a CertificateSigningRequest object.

# csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: <Your CSR Name>
spec:
  request: <base64-encoded server.csr>
  signerName: <Your signer name>
  usages:
  - digital signature
  - key encipherment
  - server auth

Here, for signer, you have the option of using custom signer or Kubernetes signer.

Then create and approve the request using the following command:

$ kubectl apply -f csr.yaml
$ kubectl approve certificate approve <Your CSR Name>

If you have the Condition field as Approved, then your ready to signed the certificate:

$ kubectl get csr
NAME                  AGE   SIGNERNAME            REQUESTOR              REQUESTEDDURATION   CONDITION
metrics-server-csr    10m   example.com/serving   yourname@example.com   <none>              Approved

Sign the request via CA

In case you used Kubernetes signer in the previous step, you can skip this step and download the signed certificate in the object status as follows:

$ kubectl get csr <csr request name> -o json | \\
  jq '.status.certificate = "'$(base64 ca-signed-server.pem | tr -d '\\n')'"' | \\
  kubectl replace --raw /apis/certificates.k8s.io/v1/certificatesigningrequests/<csr request name>/status -f -
$ kubectl get csr <csr request name> -o jsonpath='{.status.certificate}' \\
    | base64 --decode > server.crt

and then directly create the certificate secret:

$ kubectl create secret tls <secret name> --cert server.crt --key server-key.pem

If you need to create your own CA, first create a CA JSON configuration:

// ca.json
{
  "CN": "My Example Signer",
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}

$ cfssl gencert -initca ca.json |  cfssljson -bare ca
$ kubectl create configmap <CA configmap name> --from-file ca.crt=ca.pem

Now, having ca-key.pem and ca.pem then sign it using a signing configuration:

// server-signing-config.json
{
    "signing": {
        "default": {
            "usages": [
                "digital signature",
                "key encipherment",
                "server auth"
            ],
            "expiry": "876000h",
            "ca_constraint": {
                "is_ca": false
            }
        }
    }
}

$ kubectl get csr <csr request name> -o jsonpath='{.spec.request}' | \\
  base64 --decode | \\
  cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - | \\
  cfssljson -bare ca-signed-server

Finally, you’re done! download the certificate and create a secret based on that, as mentioned earlier.

Attaching certificate to services

Now that we have created our TLS secret, we only need to attach it to the target resource file. I will be attaching them using Helm files for Prometheus and Prometheus-Adapter.

Attaching the certificate

Prometheus-Adapter

Let’s start by Prometheus-Adapter

# prometheus-adapter-values.yaml
namespaceOverride: <namespace>
logLevel: 10
runAsUser: 12100
podSecurityContext:
  fsGroup: 11000
prometheus:
  url: https://<prometheus service name>.<prometheus namespace>
extraArguments:
  - --tls-cert-file=/var/run/serving-cert/tls.crt
  - --tls-private-key-file=/var/run/serving-cert/tls.key
  - --client-ca-file=/etc/ssl/certs/ca.crt
extraVolumes:
  - name: volume-serving-cert
    secret:
      secretName: <prometheus-adapter certificate secret name>
  - name: ssl-certs
    configMap:
      name: <CA configmap Name>
extraVolumeMounts:
  - mountPath: /var/run/serving-cert
    name: volume-serving-cert
    readOnly: true
  - name: ssl-certs
    mountPath: /etc/ssl/certs
    readOnly: true
resources:
  limits:
    cpu: 100m
    memory: 128Mi
    ephemeral-storage: 1Gi
rules:
  default: false
  custom:
    - seriesQuery: 'container_network_receive_packets_total{namespace!="",pod!=""}'
      resources:
        template: "<<.Resource>>"
      name:
        matches: "^(.*)"
        as: "packets-per-minutes"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

Prometheus

# prometheus-values.yaml
server:
  extraSecretMounts:
  - mountPath: /var/run/serving-cert
    name: volume-serving-cert
    secretName: <prometheus certificate secret name>
		readOnly: true
	extraConfigmapMounts:
  - name: ssl-certs
    mountPath: /etc/ssl/certs
    configMap: <CA configmap Name>
		readOnly: true
extraScrapeConfigs: |
  - job_name: myjob
    scrape_interval: 15s
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets:
          - ###.##.###.###:#####
    tls_config:
      ca_file: /etc/ssl/certs/ca.pem
      key_file: /var/run/serving-cert/key.pem
      cert_file: /var/run/serving-cert/cert.pem
      insecure_skip_verify: true

Metrics-server

Metrics-server may come by default by your Kubernetes distribution. Edit the resource manifest and add the following arguments to the container:

spec:                                                                                                                                                                                     
    containers:                                                                                                                                                                             
    - args:                                                                                                                                                                                 
      - --client-ca-file=/etc/ssl/certs/ca.pem
      - --tls-cert-file=/var/run/serving-cert/cert.pem
      - --tls-private-key-file=/var/run/serving-cert/key.pem

In addition to the argument, append the volumes to the resource, so it can access the secret and configmap:

volumeMounts:
  - mountPath: /var/run/serving-cert
    name: volume-serving-cert
    readOnly: true
  - mountPath: /etc/ssl/certs
    name: ssl-certs
    readOnly: true
volumes:
  - name: volume-serving-cert
    secret:
      secretName: <metrics-server TLS secret name>
  - configMap:
      name: <CA configmap name>
    name: ssl-certs