Effective debugging is essential to developers productivity. In addition to the obvious use case of finding bugs when their exact place is unknown or understanding why the code fails at a specific line, there are other reasons to use an interactive debugger, for example:

• Aid development: make code change ⇒ run the new code to see if it works ⇒ make more code changes ⇒ run it again, etc … — kind of “turbocharged” development mode with fast feedback to boost productivity.
• Learning the code by examining its run-time behavior, before making functional changes to it. This is especially useful if the code is complicated or/and poorly written.

Recently, while working on a few changes to the RedHat OpenShift container platform (OCP) source code, I was looking for a way to interactively step through the code to better understand how it works, and quickly try different ways of fixing it. Being not very experienced GO developer, I frequently make mistakes, so having to re-build and then re-deploy the code after each error was horribly inefficient. Ability to do code change and then immediately debug it could make a big difference.

OpenShift code can be tricky to run under the debugger. This article explains how to setup GO development environment with an interactive debugger by leveraging the Visual Studio Code feature called “Remote Development using SSH”.

The service under development will be “Cluster Network Operator” (CNO). We are going to configure VsCode to run inside CNO container.

Configuration steps at a high-level

• Build the development image that extends the CNO image and includes SSH server in addition to any other tools needed for remote coding and debugging. Some OCP services introspect container images at run time, so extending the original CNO image ensures its properties are retained.
• Make a backup copy of the original CNO deployment to restore it once development is finished.
• Change Security Context Constraints (SCC) of CNO service account to allow SSH server to run inside CNO development container.
• Patch CNO deployment to remove a few interfering features (ex. health probes), configure Security Context for CNO container to run SSH server, scale down CNO to a single replica, and replace the original CNO image with our own development image built in the previous step.
• Expose the modified CNO container with a Kubernetes node-port service and configure it with an available port. The node-port service will be used to communicate with the remote development environment using SSH channel, provided that workers IP addresses are accessible. Alternatively, use port forwarding if worker nodes are in a private network and not easily accessible.
• Configure VsCode to use an SSH channel for remote development. This can be done by adding a record for the remote host to ~/.ssh/config file.
• Once the doctored CNO deployment starts its pod, log into the CNO container and prepare development environment inside it: clone CNO source code, configure git, etc.
• Start VsCode and connect to the remote development environment.

Configuration steps can be automated

Although it is possible to manually run all the commands required to implement the steps above, it would not be practical. Luckily this simple open-source tool provides scripts to automate the process.

Clone the repository, make its scripts available to run from any directory, and update the global configuration file with personal preferences.

For CNO there are two configuration files:

• cno.conf — contains deployment name, main container name inside the deployment, as well as a unique port for SSH connection to the container.
• launch-cno.json — standard VsCode debugger configuration file.

Before starting the scripts, log into personal public quay.io registry (docker.io can also be used, but logging into quay.io is still required to get OCP images), make sure test OCP cluster is available with admin access, and then scale down the Cluster Version Operator (CVO) to 0 replicas — if the operator is left running, it immediately reverts any changes to the CNO deployment:

$ docker login quay.io -u michael_topchiev+robot -p my-robot-password
$ export KUBECONFIG=$HOME/kubeconfig/test-cluster.kubeconfig
$ oc scale deploy cluster-version-operator --replicas=0 -n openshift-cluster-version

Now setting up CNO remote development environment will take just one command with two parameters: configuration file name (minus .conf) and service namespace:

$ debug-start cno openshift-network-operator

VsCode starts and automatically connects to the CNO container. Install GO plugin and start coding.

Finding the right source code version

The operator code initially checked out in VsCode will be the master branch. We need to find the source code version that was used to build CNO for our particular OpenShift cluster release:

$ oc get clusterversion -o "jsonpath={.items[0].status.desired.image}"
    quay.io/openshift-release-dev/ocp-release@sha256:562c214e9662e3684ac5b8c3d5d2759a83f544da8897a00ebe91a02fcad6d2d9

$ oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:562c214e9662e3684ac5b8c3d5d2759a83f544da8897a00ebe91a02fcad6d2d9  --commits=true | grep cluster-network-operator
    cluster-network-operator    https://github.com/openshift/cluster-network-operator    961ca3d65ce046e29cd105e2c18563bd91dbfdae

The output shows that 961ca3d65ce046e29cd105e2c18563bd91dbfdae is the Git commit used to build CNO for the test cluster. Check the commit out inside VsCode:

$ git checkout 961ca3d65ce046e29cd105e2c18563bd91dbfdae

Once the commit is checked out, begin making code changes, set breakpoints, step through the code with the debugger, use all the features that VsCode and Git offer for coding and debugging, just like in a local development environment:

Once finished, remove the development container and restore the original CNO deployment:

$ debug-stop cno openshift-network-operator

This debugging method is not limited to OCP code

The scripts have been tested on WSL with OCP version 4.13, and can be configured to debug many other OCP operators and services. But it is not limited to working only with OpenShift code — almost any code can be developed in a remote cluster with the help of this simple tool.