Using encrypted container images in a confidential VM

Published in

ITNEXT

11 min readNov 20, 2023

Let’s say you must run your sensitive workload in a third-party infrastructure as part of your business needs, and you want to leverage a confidential computing (CC) environment to protect your intellectual property (IP) in the workload. It’s a single workload available as a container image, and if you don’t require a Kubernetes cluster, then this blog will help.

Confidential computing protects your workload from unauthorised entities — the host or hypervisor, system administrators, service providers, other VMs, and processes on the host.

In this blog, I’ll share instructions on setting up a confidential computing environment in Azure and demonstrate how to run an encrypted container image using podman.

If using encrypted container images in general with podman, please refer to the following blog for more details — https://blog.while-true-do.io/podman-encrypted-images/

The diagram below shows a high-level overview of the involved components.

Here is a brief explanation of the components.

On the infrastructure side, you have the confidential computing (CC) hardware, operating system and hypervisor with the necessary support for confidential computing.
Confidential VM — This virtual machine provides a VM-based Trusted Execution Environment (TEE). In other words, any workload running inside this VM can reap the benefits of confidential computing.
Attestation Agent (AA) — AA is responsible for starting the remote attestation process by sending cryptographic evidence of the TEE platform to the Key Broker Service (KBS).
Confidential Data Hub (CDH) and API Servier (ASR)— These components provide easy-to-use local API for retrieving secrets from Key Management Service (KMS) after successful attestation.
Key Broker Service (KBS) — KBS is the entry point for the remote attestation procedure. It works with the Attestation Service (AS) to verify the claims and the Key Management Service (KMS) to send requested secrets to the TEE.
Attestation Service (AS) — AS verifies the claims sent by AA.
Key Management Service (KMS) — KMS stores secrets that are sent to the TEE by KBS after successful attestation.

The AA, CDH and ASR components are made available by the CNCF confidential containers project. Check out this project if you plan to deploy confidential containers in a Kubernetes cluster. I have added a few relevant reading materials in the references section that you can refer to for more details.

Also, in the setup described here, the KBS, AS and KMS are deployed as a single container workload running in another confidential environment. These components are also made available by the CNCF confidential containers project.

Please note that we store the plain text secrets in the memory, and securing them using a hardware security module or other key management services is out of the scope of this blog.

I used RHEL 9.3 confidential VMs (CVM) in Azure for my test setup. You can read more about the RHEL 9.3 CVM image availability in the following blogs:

What's new: RHEL 9.3 support for AMD confidential VMs, temp disk encryption, new regions

We are thrilled to announce the RHEL 9.3 support for AMD confidential VMs, expansion of Azure Confidential VMs…

techcommunity.microsoft.com

Red Hat Enterprise Linux 9.3 on Azure confidential virtual machines: What's new?

With the release of RHEL 9.3 , RHEL CVM Preview image on Azure became available as "public preview" so no specific…

www.redhat.com

Note that to create a confidential VM using RHEL 9.3, you’ll need to use the following image details with a supported confidential instance size:

image publisher: redhat
image offer: rhel-cvm
image plan: 9_3_cvm_sev_snp

Here is the direct link to the image: https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/redhat.rhel-cvm

I used two Standard DC2as v5 instance-size VMs with RHEL 9.3 images for this blog. One RHEL confidential VM to encrypt the container image and run the KBS (and its components). And the other confidential VM to run the encrypted container image.

Generate the keys for encrypting/decrypting container images

This setup should be in an environment that you trust, for example, an on-premise environment or an environment under your control.

I used a RHEL 9.3 CVM in Azure as my test setup for this blog.

Confirm that there is no swap device

[bpradipt@kbs-cvm ~]$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.3Gi       512Mi       6.4Gi        16Mi       726Mi       6.8Gi
Swap:             0B          0B          0B

Install the required packages

[bpradipt@kbs-cvm]$ sudo dnf install -y podman skopeo jq tmux

Generate the keys for encrypting/decrypting the container image

You can use the official nginx image that comes with openssl. Or you can install openssl in your VM to generate the keys.

[bpradipt@kbs-cvm ~]$ mkdir /dev/shm/certs

[bpradipt@kbs-cvm ~]$ podman run \
  -it -v /dev/shm/certs:/certs:z \
  docker.io/nginx openssl genrsa -out /certs/key.pem

Trying to pull docker.io/library/nginx:latest…
Getting image source signatures
Copying blob 6f837de2f887 done
Copying blob e398db710407 done
Copying blob 578acb154839 done
Copying blob 85c41ebe6d66 done
Copying blob 7170a263b582 done
Copying blob 8f28d06e2e2e done
Copying blob c1dfc7e1671e done
Copying config c20060033e done
Writing manifest to image destination

[bpradipt@kbs-cvm ~]$ ls /dev/shm/certs
key.pem

[bpradipt@kbs-cvm ~]$ podman run \
  -it -v /dev/shm/certs:/certs:z \
  docker.io/nginx openssl rsa -in /certs/key.pem -pubout -out /certs/pub.pem

writing RSA key

[bpradipt@kbs-cvm ~]$ ls -l /dev/shm/certs
key.pem  pub.pem

Deploy the Key Broker Service

Let’s go through the steps to set up KBS and make the keys available for use by the confidential VM to run the encrypted container image.

Create a KBS configuration file

Details on the config parameters are available here.

[bpradipt@kbs-cvm ~]$ mkdir kbs
[bpradipt@kbs-cvm ~]$ cat <<EOF > kbs/kbs-config.toml
sockets = ["0.0.0.0:8080"]
insecure_http = true
# Path to private key for HTTPS 
#private_key = 
# Path to certificate file for HTTPS 
#certificate =
auth_public_key = "/opt/confidential-containers/kbs/user-keys/kbs-auth-pub.pem"

[as_config]
work_dir = "/opt/confidential-containers/attestation-service"
policy_engine = "opa"
rvps_store_type = "LocalFs"
attestation_token_broker = "Simple"

[as_config.attestation_token_config]
duration_min = 5
EOF

[bpradipt@kbs-cvm ~]$ ls kbs/kbs-config.toml
kbs/kbs-config.toml

You should ideally use HTTPS ( insecure_http = false ), or you should have a secure tunnel between the confidential VM running your encrypted container and the KBS system.

Generate KBS auth keys

[bpradipt@kbs-cvm ~]$ podman run \
   -it -v /dev/shm/certs:/certs:z \
   docker.io/nginx openssl genpkey -algorithm ed25519 -out /certs/kbs-auth-key.pem

Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 6f837de2f887 done
Copying blob e398db710407 done
Copying blob 578acb154839 done
Copying blob 7170a263b582 done
Copying blob 85c41ebe6d66 done
Copying blob 8f28d06e2e2e done
Copying blob c1dfc7e1671e done
Copying config c20060033e done
Writing manifest to image destination

[bpradipt@kbs-cvm ~]$ podman run \
  -it -v /dev/shm/certs:/certs:z \
  docker.io/nginx openssl pkey -in /certs/kbs-auth-key.pem -pubout -out  /certs/kbs-auth-pub.pem

[bpradipt@kbs-cvm ~]$ ls /dev/shm/certs
kbs-auth-key.pem  kbs-auth-pub.pem  key.pem  pub.pem

Start KBS

Note that the container image encryption/decryption keys which are under /dev/shm/certs are mounted under “/opt/confidential-containers/kbs/repository/default/image-decryption-keys”.

[bpradipt@kbs-cvm ~]$ podman run \
  -itd -p 8080:8080 \
  -v ./kbs/kbs-config.toml:/etc/kbs/kbs-config.toml:z \
  -v /dev/shm/certs:/opt/confidential-containers/kbs/repository/default/image-decryption-keys:z \
  -v /dev/shm/certs/kbs-auth-pub.pem:/opt/confidential-containers/kbs/user-keys/kbs-auth-pub.pem:z \
  quay.io/openshift_sandboxed_containers/kbs:dev-preview /usr/local/bin/kbs --config-file /etc/kbs/kbs-config.toml

d8eec635e49fcdae2bcec5e726793a4638b0f75861363ebec851ccda03122035

[bpradipt@kbs-cvm ~]$ podman ps
CONTAINER ID  IMAGE                                                   COMMAND     CREATED        STATUS        PORTS                    NAMES
d8eec635e49f  quay.io/openshift_sandboxed_containers/kbs:dev-preview  bash        3 seconds ago  Up 3 seconds  0.0.0.0:41849->8080/tcp  quizzical_curran

The KBS is available at http://A.B.C.D:8080 . You’ll need this when configuring the CVM to run the encrypted container image.

Creating an encrypted container image

In the following section, I’ll explain the steps to encrypt a locally generated container image having a secret.

Create an image with a secret

Here, I’ll use busybox image for demonstration and add a secret mysecret to this image.

[bpradipt@kbs-cvm ~]$ mkdir image
[bpradipt@kbs-cvm ~]$ cd image
[bpradipt@kbs-cvm image]$ cat <<EOF > Dockerfile
FROM quay.io/bpradipt/busybox
ADD ./secret /
EOF

[bpradipt@kbs-cvm image]$ echo "mysecret" > secret

[bpradipt@kbs-cvm image]$ podman build \
  -t quay.io/bpradipt/busybox:plaintext \
  -f Dockerfile .

STEP 1/2: FROM quay.io/bpradipt/busybox
Trying to pull quay.io/bpradipt/busybox:latest...
Getting image source signatures
Copying blob 92f5a971efb7 done
Copying config a416a98b71 done
Writing manifest to image destination
STEP 2/2: ADD ./secret /
COMMIT quay.io/bpradipt/busybox:plaintext
--> 7a21b830f320
Successfully tagged quay.io/bpradipt/busybox:plaintext
7a21b830f320ec33abef7b16ded95083d3b7e5092a6739c519d5c85f6668ef0c

View the plaintext container image contents

Get the container image storage location.

[bpradipt@kbs-cvm ~]$ podman info -f json | jq .store.graphRoot
"/home/bpradipt/.local/share/containers/storage"

Search for the plaintext string — mysecret.

As you can see, the secret is visible in plaintext.

[bpradipt@kbs-cvm image]$ cd /home/bpradipt/.local/share/containers/storage
[bpradipt@kbs-cvm storage]$ grep -r mysecret *
overlay/f44272466b8286bb99784c0d7d09216a90bd6ae2421c523f8dd3e013568ff5b9/diff/secret:mysecret

[bpradipt@kbs-cvm storage]$ cd
[bpradipt@kbs-cvm ~]$

Encrypt the image

As shown below, you can encrypt and push the image directly to the registry.


[bpradipt@kbs-cvm ~]$ podman push \
 --encryption-key jwe:/dev/shm/certs/pub.pem \
 quay.io/bpradipt/busybox:plaintext \
 quay.io/bpradipt/busybox:encrypted

Getting image source signatures
Copying blob 0c9e52222743 done
Copying blob 3d24ee258efc done
Copying config bdf4acc3ca done
Writing manifest to image destination

Verify if the secret is visible

You can encrypt and push the encrypted image to a local directory for verification.

[bpradipt@kbs-cvm ~]$ podman push --encryption-key jwe:/dev/shm/certs/pub.pem \
    quay.io/bpradipt/busybox:plaintext \
    dir:/home/bpradipt/image/busybox:encrypted

Search for the plaintext string — mysecret.

As you can see, the secret is not visible in the encrypted image.

[bpradipt@kbs-cvm ~]$ cd image/busybox\:encrypted/
[bpradipt@kbs-cvm busybox:encrypted]$ ls
00941a6ad9ac9318a7b1e94a37679f466044ce4d382d2a0db5b8ed018b1c38d9  7a21b830f320ec33abef7b16ded95083d3b7e5092a6739c519d5c85f6668ef0c  version
51b4e165da7bd8589316580723e3af4c95d34bd9ee8ba539ca88b8a74ab23f55  manifest.json
[bpradipt@kbs-cvm busybox:encrypted]$ grep -r mysecret *

Inspecting the encrypted image stored in the registry

You can also inspect the encrypted image in the registry by using skopeo.

The MIMEType for the layer will be “application/vnd.oci.image.layer.v1.tar+gzip+encrypted”.

[bpradipt@kbs-cvm ~]$ skopeo inspect \
  docker://quay.io/bpradipt/busybox:encrypted

{
    "Name": "quay.io/bpradipt/busybox",
    "Digest": "sha256:5ba1224c20d92dec0d9db49c415a341ec759954f64ff9775b9d47d4c4b4e6909",
    "RepoTags": [
        "latest",
        "enc",
        "encrypted"
    ],
    "Created": "2023-11-16T10:45:57.981359772Z",
    "DockerVersion": "",
    "Labels": {
        "io.buildah.version": "1.31.3"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:b027775ee059e14038cc37ecae201bc4ec44a23c0b1aecd14d57e66817ef9004",
        "sha256:7e9a27f85b68e0ebd452719ab4113a25ef8a27526039f37940e0b4384e370638"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:b027775ee059e14038cc37ecae201bc4ec44a23c0b1aecd14d57e66817ef9004",
            "Size": 2274070,
            "Annotations": {
                "org.opencontainers.image.enc.keys.jwe": "eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJNklrRXlOVFpIUTAwaWZRIiwiZW5jcnlwdGVkX2tleSI6IklUcWQxRi1XWEZ2VEV3NlhrMjRfUmsyb1dyWVlnU2lodVpBVV80bUJmVk1scFphLXc1b1FNTHhSMTVNb0pnS2dtSGM1Z0RXMDdEaXAwRkhDVEV3WUljRFppZkNtWnlERWhGTmpIZGZjbHluTmtRTm96cDM3QXB6eDdXa0VLZTBENURFci1QNkJnenZYMnFsOEIwaUJIRmRUbVVyLXBkbnkxQVhPU2xSLTRhY2NDX0lZMkNQUm9qN3NOOU9FNDJLQlM3bVg0SDVvSzZTRVYyYXU4MC1yR0ZvMXF4MHByZjNzOGhhakQ2MF9vZE1XZ1FZZHcyZVYxUWNYTnhOSlNMWEJMTlRHMXJmTVh1UzF6MXZROXo1NmZDR3dId2Z6S2s2cGRvY2lpQk9PaWpKSVFVdm1zYWp0dFZ0RWVoa2pUTkladjU0R2JUMUlnY0pVc0NEN0xZS1FJZyIsIml2IjoieTlkSllOYm9MTHB6cFcwaiIsImNpcGhlcnRleHQiOiJlbUhFamNjaFNhcktLRnZ0TUJya2hsWXVhakR4YzBobXhaZ05LUHRmUmhNeTRSbDluQUFNNEl2cVQxZEMwdjRYNTQtQUZfOVlIT3lsY3dXV0JRYnFZSU5wSHNWdk5uMzhXNTJBamRheHZfVWxfYl9OT251RTM1cmVnYnRXOThYbUp4QXhVZEx2ZGd5TGJiQzk0Y1V2dklFWWhjSXFMd0hoeHJrRVFOZkR3TW9hUFpBVmh3cjcwbkNFaDBIb1NYcXZuakktRU1YaEI4bVFrN0MtUWRNZEpHN3FVVG5Cd3RyejdzYmExS0I0c216MlVWUjV5a3lOVFdYNWpHanZ3YjZZLWciLCJ0YWciOiJxcTZEb1cyVVZMbFo0TnYzNnB1dnJ3In0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJYK2x3YWVnNlV6dnV3T0FKaU9kVnltN1NrNUlWbXByR0M0a1ZwUXZQT1ZVPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:7e9a27f85b68e0ebd452719ab4113a25ef8a27526039f37940e0b4384e370638",
            "Size": 112,
            "Annotations": {
                "org.opencontainers.image.enc.keys.jwe": "eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJNklrRXlOVFpIUTAwaWZRIiwiZW5jcnlwdGVkX2tleSI6IlJmYlJqd2NZX2ozWDJ1N1B1UXhtckdPU0lRY29QRmxSOGY4NWdHY2JVRE9VMHNmcHFvbFFidXJqOWFlbGlOeHdRMUYteEx3Zk9xMEZSWkx0TXNoOXZCQVpXVlNZTmxXTzlEZ1MxMlcwbWI1UkswX2lfTGNMT1EwakRoX3pXVnBmd0RGTnRzME1naXNaSldZWVRJazZ6UzNlWEdNQXBBakNaN2I5MnlxbVNZeEl3YnVlaFdUbWZ3bjZhbkVnWjhQZHowRU5mVTV0WVRfMDF3QzNDV0pmem1yZExVMnBBaE83ekFLV09oQ3VGZmZ0SXJnajd0Zmc5ZW12SzNfVTQ1WHkxM1VEUVFPTGFfVnZhQzNMX0xjd2l1Z2VjOGgxWndDaWxxOVBGbkRIQUhvcXpHbXktTkQ3eThlQmZ2MDdUTklYY2VMYmhQZldzeHlFSGZCWjR1N3UwUSIsIml2IjoiUzlYa0RaYWFtS1dENnZqUCIsImNpcGhlcnRleHQiOiJ6aWpzSVFpenFxR1ZlLXdhb1JwQ3d5M3ZFeTUtQTEwSXJlcWFaRlZKTjg5NGVvQTVoS2h6M3BrbjNfVkluaG5GZWFwbmV4bnVMVVlZQ09GYVA4N0lDZG1HSUVCTnZMRFhKejBsVHN4MjJmTU9xTzVza3FnS3RmalBiZHRiQjFvcy1NR1pOMmQxUjhMdU15RldaRDZSTFpqNjExUURocG5rcHoySkZTbEctVXlETlZSMEtPMUVMejVadlFyUktlZ25Hdi1ZRGtGRmVHcXF5R09MWFFLZGVoTUlSUlJrYlpRazhtWjhUWHREbGJEcGVYYXVXcFQ5NnBTN21wSk55clh2YkEiLCJ0YWciOiJBTzdRQmVSRE5hWURyeTR1QVJMeFd3In0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiI2Z25xL04xeVM5Um1vdS9xSXZlUHhYVXdqekl1YWl3VExOajB6TXViRHhJPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

Let’s look at the org.opencontainers.image.enc.keys.jwe annotation:

[bpradipt@kbs-cvm ~]$ echo "eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJNklrRXlOVFpIUTAwaWZRIiwiZW5jcnlwdGVkX2tleSI6IlJmYlJqd2NZX2ozWDJ1N1B1UXhtckdPU0lRY29QRmxSOGY4NWdHY2JVRE9VMHNmcHFvbFFidXJqOWFlbGlOeHdRMUYteEx3Zk9xMEZSWkx0TXNoOXZCQVpXVlNZTmxXTzlEZ1MxMlcwbWI1UkswX2lfTGNMT1EwakRoX3pXVnBmd0RGTnRzME1naXNaSldZWVRJazZ6UzNlWEdNQXBBakNaN2I5MnlxbVNZeEl3YnVlaFdUbWZ3bjZhbkVnWjhQZHowRU5mVTV0WVRfMDF3QzNDV0pmem1yZExVMnBBaE83ekFLV09oQ3VGZmZ0SXJnajd0Zmc5ZW12SzNfVTQ1WHkxM1VEUVFPTGFfVnZhQzNMX0xjd2l1Z2VjOGgxWndDaWxxOVBGbkRIQUhvcXpHbXktTkQ3eThlQmZ2MDdUTklYY2VMYmhQZldzeHlFSGZCWjR1N3UwUSIsIml2IjoiUzlYa0RaYWFtS1dENnZqUCIsImNpcGhlcnRleHQiOiJ6aWpzSVFpenFxR1ZlLXdhb1JwQ3d5M3ZFeTUtQTEwSXJlcWFaRlZKTjg5NGVvQTVoS2h6M3BrbjNfVkluaG5GZWFwbmV4bnVMVVlZQ09GYVA4N0lDZG1HSUVCTnZMRFhKejBsVHN4MjJmTU9xTzVza3FnS3RmalBiZHRiQjFvcy1NR1pOMmQxUjhMdU15RldaRDZSTFpqNjExUURocG5rcHoySkZTbEctVXlETlZSMEtPMUVMejVadlFyUktlZ25Hdi1ZRGtGRmVHcXF5R09MWFFLZGVoTUlSUlJrYlpRazhtWjhUWHREbGJEcGVYYXVXcFQ5NnBTN21wSk55clh2YkEiLCJ0YWciOiJBTzdRQmVSRE5hWURyeTR1QVJMeFd3In0=" | base64 -d | jq
{
  "protected": "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
  "encrypted_key": "RfbRjwcY_j3X2u7PuQxmrGOSIQcoPFlR8f85gGcbUDOU0sfpqolQburj9aeliNxwQ1F-xLwfOq0FRZLtMsh9vBAZWVSYNlWO9DgS12W0mb5RK0_i_LcLOQ0jDh_zWVpfwDFNts0MgisZJWYYTIk6zS3eXGMApAjCZ7b92yqmSYxIwbuehWTmfwn6anEgZ8Pdz0ENfU5tYT_01wC3CWJfzmrdLU2pAhO7zAKWOhCuFfftIrgj7tfg9emvK3_U45Xy13UDQQOLa_VvaC3L_Lcwiugec8h1ZwCilq9PFnDHAHoqzGmy-ND7y8eBfv07TNIXceLbhPfWsxyEHfBZ4u7u0Q",
  "iv": "S9XkDZaamKWD6vjP",
  "ciphertext": "zijsIQizqqGVe-waoRpCwy3vEy5-A10IreqaZFVJN894eoA5hKhz3pkn3_VInhnFeapnexnuLUYYCOFaP87ICdmGIEBNvLDXJz0lTsx22fMOqO5skqgKtfjPbdtbB1os-MGZN2d1R8LuMyFWZD6RLZj611QDhpnkpz2JFSlG-UyDNVR0KO1ELz5ZvQrRKegnGv-YDkFFeGqqyGOLXQKdehMIRRRkbZQk8mZ8TXtDlbDpeXauWpT96pS7mpJNyrXvbA",
  "tag": "AO7QBeRDNaYDry4uARLxWw"
}

Now that the encrypted image is available in the public registry — quay.io, let's use it in a confidential environment.

Using the encrypted image

These steps are executed in another Azure RHEL 9.3 CVM environment using the Standard DC2as v5 instance size.

Confirm that there is no swap device

[azureuser@cvm ~]$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.3Gi       512Mi       6.4Gi        16Mi       726Mi       6.8Gi
Swap:             0B          0B          0B

Install the required packages

$ sudo dnf install -y podman jq tmux

Download Attestation Agent (AA), Confidential Data Hub (CDH) and API server (ASR) components

[azureuser@cvm ~]$ mkdir binaries && cd binaries

[azureuser@cvm binaries]$ podman run \
  -it --rm -v $PWD:/out:z \
  quay.io/openshift_sandboxed_containers/openshift-sandboxed-containers-payload:rhel93snp cp /podvm-binaries.tar.gz /out/

[azureuser@cvm binaries]$ ls
podvm-binaries.tar.gz

[azureuser@cvm binaries]$ tar xzvf podvm-binaries.tar.gz \
--strip-components=3 \
usr/local/bin/attestation-agent \
usr/local/bin/confidential-data-hub \
usr/local/bin/api-server-rest

[azureuser@cvm binaries]$ ls
api-server-rest attestation-agent confidential-data-hub podvm-binaries.tar.gz

[azureuser@cvm binaries]$ sudo cp api-server-rest attestation-agent confidential-data-hub /usr/local/bin

[azureuser@cvm binaries]$ cd

Create required configuration files for AA, CDH, and ASR

Note the KBS_ADDR. This is the address of the KBS, i.e., http://KBS_IP:8080 in my current setup.

[azureuser@cvm ~]$ sudo mkdir -p /peerpod
[azureuser@cvm ~]$ sudo touch /peerpod/daemon.json

[azureuser@cvm ~]$ export KBS_ADDR="http://KBS_IP:8080"
[azureuser@cvm ~]$ export AA_KBC_PARAMS="cc_kbc::$KBS_ADDR"
[azureuser@cvm ~]$ sudo bash -c "cat <<EOF > /etc/agent-config.toml
aa_kbc_params = \"$AA_KBC_PARAMS\"
EOF
"
[azureuser@cvm ~]$ cat /etc/agent-config.toml
aa_kbc_params = "cc_kbc::http://KBS_IP:8080"

Create a bash script to start the AA, CDH, and ASR programs

[azureuser@cvm ~]$ cat <<EOF > start_attesation_progs.sh
#!/bin/bash
# Start a new tmux session named "my_session"
tmux new-session -d -s my_session
# Split the tmux window vertically
tmux split-window -v
# Split the right pane horizontally
tmux split-window -h
# Select the first pane and start program 1
tmux select-pane -t 0
tmux send-keys "/usr/local/bin/attestation-agent" C-m
# Select the second pane and start program 2
tmux select-pane -t 1
tmux send-keys "/usr/local/bin/confidential-data-hub" C-m
# Sleep for few seconds to let the other programs start
sleep 5
# Select the third pane and start program 3
tmux select-pane -t 2
tmux send-keys "/usr/local/bin/api-server-rest --features all" C-m
# Attach to the tmux session to view the programs
tmux attach-session -t my_session
EOF

Start the AA, CDH and ASR programs

[azureuser@cvm ~]$ chmod +x start_attesation_progs.sh
[azureuser@cvm ~]$ sudo ./start_attesation_progs.sh

You can create systemd-units or use ansible to configure the environment.

Please note that these programs are not officially supported as part of RHEL support.

Get the decryption key

The key is available from the KBS at “/opt/confidential-containers/kbs/repository/default/image-decryption-keys/key.pem”.

The key is copied into memory — /dev/shm/key.pem .

You can use curl to download the decryption key from KBS, as shown below. The key will be available only after successful attestation.

Note that AA, CDH and ASR measurements are currently not used.

[azureuser@cvm ~]$ curl -L http://127.0.0.1:8006/cdh/resource/default/image-decryption-keys/key.pem -o /dev/shm/key.pem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1704 100 1704 0 0 594 0 0:00:02 0:00:02 - : - : - 594

Setup podman to use memory as the backing store for container images

Ensure the container storage is backed by memory so you don’t accidentally leak the secret info in the container image by writing to an unencrypted disk.

You can check the storage location in the following file — /etc/containers/storage.conf .

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[azureuser@cvm ~]$ sudo mount -t tmpfs tmpfs $HOME/.local/share/containers/storage

[azureuser@cvm ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           3.7G   92K  3.7G   1% /dev/shm
tmpfs           1.5G   17M  1.5G   2% /run
/dev/sda3        32G  1.4G   29G   5% /
/dev/sda2       252M   58M  194M  23% /boot/efi
tmpfs           747M   32K  747M   1% /run/user/1000
tmpfs           3.7G     0  3.7G   0% /home/bpradipt/.local/share/containers/storage

If you are using the root user to run the containers, ensure you enable transient store by uncommenting # transient_store=true and mount “/var/lib/containers/storage” to memory.

transient_store = true

[azureuser@cvm ~]$ sudo mount -t tmpfs tmpfs /var/lib/containers/storage

Download and run the encrypted container image

As shown below, running the image without the key fails.

[azureuser@cvm ~]$ podman run -it quay.io/bpradipt/busybox:encrypted

Trying to pull quay.io/bpradipt/busybox:encrypted...
Getting image source signatures
WARN[0000] Compressor for blob with digest sha256:7e9a27f85b68e0ebd452719ab4113a25ef8a27526039f37940e0b4384e370638 previously recorded as gzip, now uncompressed
Copying blob 7e9a27f85b68 done
Copying blob b027775ee059 done
Error: writing blob: adding layer with blob "sha256:b027775ee059e14038cc37ecae201bc4ec44a23c0b1aecd14d57e66817ef9004": processing tar file(archive/tar: invalid tar header): exit status 1

Download and run the encrypted container image with the decryption key

As shown below, the container runs successfully, and the secret — mysecret is available inside the confidential VM.

[azureuser@cvm ~]$ podman run -it \
  --decryption-key /dev/shm/certs/key.pem \
  quay.io/bpradipt/busybox:encrypted

Trying to pull quay.io/bpradipt/busybox:encrypted...
Getting image source signatures
WARN[0000] Compressor for blob with digest sha256:7e9a27f85b68e0ebd452719ab4113a25ef8a27526039f37940e0b4384e370638 previously recorded as uncompressed, now gzip
Copying blob 7e9a27f85b68 done
Copying blob b027775ee059 done
Copying config 7a21b830f3 done
Writing manifest to image destination
/ # cat /secret
mysecret

Conclusion

I wanted to demonstrate the simplicity of using a confidential computing environment. The use of confidential computing technology can unlock new business opportunities, and there are now multiple ways to achieve the benefits of confidential computing.

The software components used in this blog are common for confidential VMs and containers and are being developed under the CNCF confidential containers project. I hope that you find this blog helpful in trying out your confidential computing use cases.

References

Listing down a few additional reading materials which will be helpful in understanding the technology behind confidential virtual machines and confidential containers.

RHEL confidential virtual machines in Azure — technical deep dive
Confidential containers in Kubernetes
Confidential containers in Azure — technical deep dive
Understanding Remote Attestation
From sandboxed containers to confidential containers — part-1 and part-2