Securing Kubernetes Workloads: A Practical Approach to Signed and Encrypted Container Images

Podman — one tool to rule them all

Published in

ITNEXT

7 min readFeb 19, 2024

If you are reading this, you most likely have experienced and also appreciate the portability brought about by container images. The ease of packaging applications and their dependencies consistently has a profound impact on the entire software supply chain. However, this very portability and ease of packaging also raises a concern about the trustworthiness of the container images.

Signed container images address this challenge by implementing cryptographic signatures, binding the image to a specific entity, thereby creating a chain of trust. It enables verification of the authenticity and origin of the container image. This process ensures that the image is not tampered with and comes from a legitimate source. Without signed images, there is a heightened risk of deploying containers having malicious code or vulnerabilities, compromising the entire IT environment.

Encrypted container images further enhance the security posture of containerised applications. It provides confidence in using public container registries for sensitive or proprietary code.

Signed and encrypted container images play an essential role in safeguarding the integrity of the software supply chain. Such images also make auditing and compliance easier. It helps organisations to track and verify the origins and changes made to the container images over time and enforce regulatory requirements.

I have said enough about the benefits of using signed/encrypted container images. How do you create and use a signed and encrypted container image?

In this article, I’ll focus on a single tool — podman, that can provide a consistent framework to use signed and encrypted container images across your environments — whether using standalone systems or Kubernetes clusters.

Current approaches for using signed container images

As part of your DevOps processes, you can sign images during the step where the image is built and pushed to the container registry. Once the images are signed, you can enforce a policy that requires all images deployed to your Kubernetes cluster to have a valid signature. This means that if there is an attempt to deploy an image that either isn’t signed or doesn’t have a valid signature matching your policy, that container image will not be deployed.

Sigstore Cosign is a popular tool for signing container images. There are other tools like, Docker Content Trust, that you can use. Standalone container engines like podman or docker have the necessary support built-in to verify signed images.

The following article describes verifying signed container images in a Kubernetes cluster. Also, cluster-wide policy controllers, like Kyverno, can help verify signed container images in the Kubernetes cluster.

In this article, I’ll focus on using podman for image signing and verification for either a standalone system or a Kubernetes cluster.

Using signed container images with podman on a standalone system

Podman supports signing with GPG and cosign keys. I’ll use cosign keys here.

On Linux, you can download the cosign binary from the release page — https://github.com/sigstore/cosign/releases/tag/v2.2.3

The first step is to create a cosign key pair.

The following command generates a key pair — cosign.pub and cosign.key

$ cd $HOME
$ mkdir certs
$ cd certs
$ cosign generate-key-pair
Enter password for private key:
Enter password for private key again:
Private key written to cosign.key
Public key written to cosign.pub

For signing images, add the following configuration file- /etc/containers/registries.d/default.yaml

docker:
 quay.io:
 use-sigstore-attachments: true

Create a file to keep the passphrase. This helps to automate the signing process without entering the passphrase manually.

Replace it with the actual passphrase if you have used one when generating the key pair. For this article I have used an empty password.

$ echo ""> certs/pass

Let’s build a sample container image for testing.

$ cd $HOME
$ mkdir image
$ cd image
$ cat <<EOF > Dockerfile
FROM quay.io/bpradipt/busybox
RUN echo "my test image" > /image-description
EOF


$ podman build \
 -t quay.io/bpradipt/busybox \
 -f Dockerfile .

Create and push the signed image to the registry.

$ podman push --sign-by-sigstore-private-key ./certs/cosign.key --sign-passphrase-file ./certs/pass quay.io/bpradipt/busybox
quay.io/bpradipt/busybox:signed

Copying blob 3d24ee258efc done |
Copying config a416a98b71 done |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures

Next, let’s verify the signature.

You’ll need to create a podman container policy for verification with the path to the public key. An example policy.json file (path:/etc/containers/policy.json ) is shown below:

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "quay.io": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/certs/cosign.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ]
        }
    }
}

Now, let’s try to pull a non-signed image. It will fail.

$ podman pull quay.io/bpradipt/busybox

Trying to pull quay.io/bpradipt/busybox:latest...
Error: Source image rejected: A signature was required, but no signature exists

Now, let’s try to pull the signed image. It will succeed.

$ podman pull quay.io/bpradipt/busybox:signed

Trying to pull quay.io/bpradipt/busybox:signed...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 702a604e206f skipped: already exists
Copying config a416a98b71 done   |
Writing manifest to image destination
Storing signatures
a416a98b71e224a31ee99cff8e16063554498227d2b696152a9c3e0aa65e5824

We can use the same approach with a Kubernetes cluster. The primary reason is to provide a consistent framework that works across different environments. The following section describes the steps for a Kubernetes cluster.

Using signed container images with podman on a Kubernetes cluster

Create a Kubernetes configmap with the policy.json file having the following contents:

$ cat $(pwd)/policy.json

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "quay.io": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/certs/cosign.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ]
        }
    }

}

# Create the configmap
$ kubectl create configmap --from-file=$(pwd)/policy.json policy-cm

Create a Kubernetes secret containing the cosign.pub file for signature verification

$ kubectl create secret generic my-secret --from-file=$(pwd)/certs/cosign.pub

Create another Kubernetes configmap having file default.yaml with the following contents:

$ cat $(pwd)/default.yaml

docker:
    quay.io:
        use-sigstore-attachments: true



#Create the configmap
$ kubectl create cm --from-file=$(pwd)/default.yaml reg-default-cm

Deploy the container using the following pod spec:

$ cat $(pwd)/sign-pod.yaml

apiVersion: v1
kind: Pod
metadata:
 name: sign-pod
spec:
 initContainers:
  - name: copy-secret
    image: busybox:latest
    command: ['sh', '-c', 'cp /secrets/cosign.pub /certs/cosign.pub']
    volumeMounts:
    - name: secret-volume
      mountPath: /secrets
    - name: certs
      mountPath: /certs
 containers:
   - name: run-image
     image: quay.io/podman/stable
     command: ["sh", "-c"]
     args:
     -  podman run quay.io/bpradipt/busybox:signed sleep 1000000 
     securityContext:
       runAsUser: 1000      
       privileged: true
     volumeMounts:
     - name: certs
       mountPath: /certs
     - name: policy-cm
       mountPath: /etc/containers/policy.json
       subPath: policy.json
     - name: reg-cm
       mountPath: /etc/containers/registries.d/
 volumes:
 - name: certs
   emptyDir:
     medium: Memory
 - name: secret-volume
   secret:
     secretName: my-secret
 - name: policy-cm
   configMap:
     name: policy-cm
 - name: reg-cm
   configMap:
     name: reg-default-cm



# Create the pod
$ kubectl apply -f $(pwd)/sign-pod.yaml

I showed one of the possibilities. You can be creative based on your requirements. For example, you can use a central repository for the required policies and the public keys for every environment. Also you can deploy the pod as a Kata container by adding suitable runtimeClassName .

Now let’s turn our attention to using encrypted container images.

Current approaches for using encrypted container images

Skopeo and podman have support for encrypting container images. You can find an excellent article here describing the process using skopeo.

Podman also supports the use of encrypted container images.

For a Kubernetes cluster, both containerd and cri-o supports using encrypted container images. The keys need to be available on the worker node.

For a Kubernetes cluster, containerd and cri-o support using encrypted container images. The keys need to be available on the worker node.

This model of having the keys available on the worker node is also called the “node” key model. You can read more about it in the following docs:

Let’s see how we can use podman to create and use encrypted container images across different environments.

Using encrypted container images with podman on a standalone system

The first step is to create the keys for encrypting the image.

$ mkdir certs

$ podman run \
 -it -v $(pwd)/certs:/certs:z \
 docker.io/nginx openssl genrsa -out /certs/key.pem

$ podman run \
 -it -v $(pwd)/certs:/certs:z \
 docker.io/nginx openssl rsa -in /certs/key.pem -pubout -out /certs/pub.pem

Let’s build a sample image containing a secret.

$ mkdir image
$ cd image
$ cat <<EOF > Dockerfile
FROM quay.io/bpradipt/busybox
ADD ./secret /
EOF

$ echo "mysecret" > secret

$ podman build \
 -t quay.io/bpradipt/busybox:plaintext \
 -f Dockerfile .

Now encrypt and push the image using the following command.

$ podman push \
--encryption-key jwe:$(pwd)/certs/pub.pem \
quay.io/bpradipt/busybox:plaintext \
quay.io/bpradipt/busybox:encrypted

You can use the following steps to verify the encrypted container image.

Note that the encrypted image is placed under $HOME/image/busybox:encrypted.

$ podman push --encryption-key jwe:$(pwd)/certs/pub.pem \
   quay.io/bpradipt/busybox:plaintext \
   dir:$HOME/image/busybox:encrypted

# Grep should return empty
cd $HOME/image/busybox\:encrypted/
grep -r mysecret *

You can run the image by specifying the decryption key as shown below:

$ podman run --decryption-key $(pwd)/certs/key.pem quay.io/bpradipt/busybox:encrypted echo "hi"

Using encrypted container images with podman on a Kubernetes cluster

Create a Kubernetes secret object containing the decryption key.

$ kubectl create secret generic my-secret --from-file=$(pwd)/certs/key.pem

Deploy a pod using the encrypted image

$ cat $(pwd)/enc-pod.yaml

apiVersion: v1
kind: Pod
metadata:
 name: enc-pod
spec:
 initContainers:
  - name: copy-secret
    image: busybox:latest
    command: ['sh', '-c', 'cp /secrets/key.pem /certs/key.pem']
    volumeMounts:
    - name: secret-volume
      mountPath: /secrets
    - name: certs
      mountPath: /certs
 containers:
   - name: run-image
     image: quay.io/podman/stable
     command: ["sh", "-c"]
     args: 
     - podman run --decryption-key /certs/key.pem quay.io/bpradipt/busybox:encrypted sleep 1000000
     securityContext:
       runAsUser: 1000
       privileged: true 
     volumeMounts:
     - name: certs
       mountPath: /certs
 volumes:
 - name: certs
   emptyDir:
     medium: Memory
 - name: secret-volume
   secret:
     secretName: my-secret


# Create the pod
$ kubectl apply -f $(pwd)/enc-pod.yaml

I have used initContainer to demonstrate one of the possible approaches for key retrieval. Currently, we copy the key from the Kubernetes secret to a shared ephemeral volume. Alternatively you can also download the key from an external Key Management Services (KMS) and copy them to the shared ephemeral volume.

Conclusion

As you can see, podman provides a baseline framework for using signed and encrypted images on a standalone system or a Kubernetes cluster.

Further, using podman allows using per-pod keys, which are more powerful and flexible than the node-level keys.

The challenge is wrapping the actual container to run with podman. You can either add podman to your container image and modify the entrypoint accordingly. Or you can use a mutating webhook to make necessary changes to the pod specification.

In next blog we’ll see how the same concept of using podman can be extended to confidential containers.