Jenkins + k8s + Buildkit: life behind the corporate proxy

In the previous article, we discussed the options available to you when it comes to building the Docker images after Kubernetes removes Dockershim in v1.24.

But what if your on-premise Kubernetes cluster has no direct access to the internet and is forced to go through the proxy server. This scenario is fairly common in the corporate world.

Also, what is common in this set up is when the self hosted Docker registry is used and in some cases, internal self-hosted resources use self-signed certificates or certificates signed by the locally created untrusted Certificate Authority. Is it possible to use Buildkit in this set up?

And the answer is yes, it is possible, but it requires a little bit extra effort to set it up.

Self-Hosted Docker Registry

Let’s discuss each scenario, starting with self-hosted Docker Registry.

No changes are required if self-hosted Docker Registry uses SSL Certificate signed by the recognized and trusted Certificate Authority.

However, if it is signed by the untrusted CA or it uses self-signed certificate it must be added to the moby/buildkit image. Since we use buildkit in a daemon mode, it must be added to the image before starting the container.

To do so, you need to create a new image extending moby/build placing you CA certificate or self-signed certificate into /usr/local/share/ca-certificates/ directory (moby/buildkit is based on alpine distro) and running update-ca-certificates command.

For example, Dockerfile will look like this:

Once you build it docker build -t myorg/buildkit .and push it to the registry docker push myorg/buildkit it is ready to be used in the pipeline. Just replace moby/buildkit in the Pod Template with the name of the newly built image and that is it:

If you push the image into the private registry that requires authentication to access images you have to create a new secret of type kubernetes.io/dockerconfigjson in the namespace that is used by Jenkins.

To do so, first encode the following JSON with the registry credentials:

The content of the “auth” property is the base64 encoded username:password string.

Encode above JSON with base64 encoding and add it as a .dockerconfigjson property in the following secret:

Run kubectl apply -f secret.yaml to create secret in the namespace and then add its name to the Pod Template’s imagePullSecrets array property:

Kubernetes Cluster Behind Proxy

If your Kubernetes Cluster is deployed behind the proxy and worker nodes do not have direct access to the internet to pull images and/or pull dependencies there are two additional settings are required.

In order to allow BuildKit to pull images if Kubernetes is behind proxy add HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables to the pod in question. In this case container definition will look like this:

In order to allow BuildKit to pull dependencies or any other network resources while building image add same environment variables to the build command. In this case build command will look like this: