Debug a Go Application in Kubernetes from IDE

As a Developer, it is always useful to be able to debug an application with its own IDE.

When your application only works with the Kubernetes API, you can simply launch your application in the IDE and connect it to the remote Kubernetes API.

But When your application needs to connect to other systems which are only available inside the Kubernetes cluster, this solution does not work anymore.

Build the application

The application I want to debug is a Cassandra Operator which is based on the CoreOS Operator SDK

The Operator used the script build.sh to build the Go application, I added an input parameter DEBUG to the script so that it can build a debug version of the application, with the addition of specific gcflags, and then suffix the binary with -debug.

Note that we also added the dlv binary (which is the Go debugger) to our target binaries.

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

if ! which go > /dev/null; then
   echo "golang needs to be installed"
   exit 1
fi

BIN_DIR="$(pwd)/tmp/_output/bin"
mkdir -p ${BIN_DIR}
PROJECT_NAME="cassandra-operator"
REPO_PATH="gitlab.si.francetelecom.fr/kubernetes/cassandra-operator"
BUILD_PATH="${REPO_PATH}/cmd/${PROJECT_NAME}"

if [ $# -gt 0 ] && [ "$1" = "DEBUG" ] ; then
  echo "building "${PROJECT_NAME}" In DEBUG Mode..."
  GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -gcflags "-N -l" -o ${BIN_DIR}/${PROJECT_NAME}-debug $BUILD_PATH
  cp /usr/local/bin/dlv ${BIN_DIR}
else
  echo "building "${PROJECT_NAME}"..."
  GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ${BIN_DIR}/${PROJECT_NAME} $BUILD_PATH
fi

Build the Docker Image

The Operator SDK generate a Dockerfile to build the image for our operator, and a script (docker_build.sh) used to build it, on which we also add the DEBUG parameter :

#!/usr/bin/env bash

if ! which docker > /dev/null; then
   echo "docker needs to be installed"
   exit 1
fi

: ${IMAGE:?"Need to set IMAGE, e.g. gcr.io/<repo>/<your>-operator"}

if [ $# -gt 0 ] && [ "$1" = "DEBUG" ] ; then
  echo "building container ${IMAGE} in DEBUG Mode..."
  docker build -t "${IMAGE}" -f tmp/build/Dockerfile-debug .
else
  echo "building container ${IMAGE}..."
  docker build -t "${IMAGE}" -f tmp/build/Dockerfile .
fi

And we add a new Dockerfile-debug with only those Modifications at the end :

....
ADD tmp/_output/bin/cassandra-operator-debug /usr/local/bin
ADD tmp/_output/bin/dlv /usr/local/bin

EXPOSE 40000

ENTRYPOINT ["/usr/local/bin/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "/usr/local/bin/cassandra-operator-debug"]

We add the debug version of the application and the delve debugger inside the docker image. We also changed the entrypoint telling the image to start the debugger which will then execute the operator in debug mode.

Delve exposes the port 40000, on which we will configure our IDE to communicate With.

Excerpt of the Makefile used to build the whole operator in debug mode :

docker-build-debug: docker-get-deps
  echo "Generate CRD Client"
  tmp/codegen/update-generated.sh
  echo "Build Go Application In DEBUG Mode"
  docker run --rm -v $(PWD):$(WORKDIR):rw $(REPOSITORY)/dev:$(VERSION) /bin/bash -c './tmp/build/build.sh DEBUG'
  echo "Build Docker Image With DEBUG Enabled"
  IMAGE=$(REPOSITORY):$(VERSION)-debug ./tmp/build/docker_build.sh DEBUG

Deploy the Application

Once you have compiled the operator in debug mode and recreate the Docker Image, we also need a small change in the deployment in order to expose the port 40000 and to add the SYS_PTRACE capability to the operator Pod in the cluster. I used a Helm chart in order to deploy the application which manages a debug value to customize the deployment :

...
{{- if .Values.debug }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}-debug"
{{- else}}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
{{- end }}
        imagePullPolicy: "{{ .Values.image.pullPolicy }}"
        resources:
{{ toYaml .Values.resources | indent 10 }}
        env:
          - name: WATCH_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 9710
          name: metrics
          protocol: TCP
{{- if .Values.debug }}
        - containerPort: 40000
          name: debug
          protocol: TCP
      securityContext:
        capabilities:
          add:
            SYS_PTRACE
{{- end }}

To deploy the operator in debug mode I just need to surcharge the debug value :

helm install ./helm/cassandra-operator --name cassandra-operator-debug --set debug=true

This will create all necessary Kubernetes objects for the operator. If you have a TCP ingress you can configure a specific route to reach the port 40000 of your pod. Since I don’t have one, I will create a port-forward to be able to reach this port from my local machine where my IDE sits:

kubectl port-forward <cassandra-operator-debug-pod-name> 40000:40000

Configure the IDE

I used Goland to debug my Go programs, but it will work similarly with others IDE

We need to create a Remote Debug and we point it to localhost:40000 since I have activated the port forward.

From that point, we are able to set Breakpoint in our local IDE, and it will communicate with the delve debugger deployed in the Kubernetes cluster to allows to debug our application :

Finally

The application will wait for your IDE to connect to it before starting execution of code, that way you are able to start debugging from start.

The Pod will stop in the state completed each time you stop debugging on IDE side (or loose connection) and may be automatically restarted if you’re using a Kubernetes deployment.

You need to know that :

• your remote debugging will be slower due to network latency between delve and your IDE
• Each time you want to test a new line of code you will need to rebuild the code in debug mode, re-build the Docker Image in debug mode, re-push it to your docker registry, then re-deploy the operator in debug mode.