Red Hat® Enterprise Linux CoreOS (RHCOS) is the immutable container host version of Red Hat Enterprise Linux (RHEL) and features a RHEL kernel with SELinux enabled by default. It includes the kubelet (which is the Kubernetes node agent) and the CRI-O container runtime (which is optimized for Kubernetes). Operating system updates are delivered as an Atomic OSTree repository where operating system changes are made in-place as an atomic operation by using rpm-ostree . Debugging kernel issues in this environment is challenging. One of the essential aspects of kernel debugging is to be able to boot into a patched kernel but this is not straightforward in a RHCOS environment as additional packages cannot be installed the traditional way. Also, most of the system directories, such as /boot and /lib are configured to be read-only.
Though kernel issues are rare in a RHCOS environment, as it is based on already available RHEL releases, there could still be exceptions. Also, kernel debugging might be needed at times for performance improvement purposes. In this blog, we explore the steps needed to compile and boot into a custom kernel on a RHCOS system. Note that privileged access is needed for most of the following steps:
- Change the current directory to a writable directory (for example, /var/home/core directory) on the RHCOS host system.
# export DIRPATH=/var/home/core
# cd $DIRPATH
- Set up a container to build a custom kernel and also to source the binary for kexec utility from it. To set up the container for building the kernel, you need to add the toolchain packages to the container image. The following output shows the contents of the container file to be used for setting up the container.
# cat Dockerfile
RUN dnf install -y kexec-tools make gcc git openssl-devel bison flex autoconf automake bc
- Use the container file defined in the previous step to create a container.
# podman build -t fedora:build -f Dockerfile
- Copy the stock kernel's config file to use it as the base for building the custom kernel.
# cp /lib/modules/`uname -r`/config .
- Start the container with the writable directory from host, bind mounted into the container, for sharing data between the RHCOS host and the container.
# podman run -it -v $DIRPATH:/root --privileged=true -- fedora:build bash
- Change to the directory that is bind mounted into the container.
# cd /root
- Clone the linux kernel source and build the kernel inside the container.
# git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
# cp config linux/.config
# cd linux
# make oldconfig
# make vmlinux; make modules
- Copy the compiled kernel and its modules into the directory that is bind mounted into the container. Also, copy the kexec utility to use it for loading and booting into the custom kernel.
# make INSTALL_MOD_STRIP=1 INSTALL_MOD_PATH=/root modules_install
# cp vmlinux /root/vmlinux-<kver>
# cp /sbin/kexec /root
- Exit the container.
- On the RHCOS host, change to $DIRPATH where the kernel image and modules are copied to. Build the initial RAM disk (initrd) image for the custom kernel by picking the dracut options from the RHCOS initrd image in /boot/ostree and use the same options with --kmoddir '$DIRPATH/lib/modules/5.9.0-rc5 to find the modules, as kernel modules could not be installed at the usual place (the /lib/modules directory is read-only).
# cd $DIRPATH
# mkdir /tmp/dracut
# dracut --reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f --add 'ignition' --no-hostonly --add-drivers 'mptspi vmw_pvscsi' --omit-drivers 'nouveau' --omit 'nfs' --add 'iscsi' --add 'ifcfg' --add 'fips' --kmoddir '$DIRPATH/lib/modules/<kver>' initramfs-<kver>.img <kver>
Note: Use the following command to find the options used for building the initrd for stock kernel.
# lsinitrd -f /usr/lib/dracut/build-parameter.txt <boot-dir>/initramfs-4.18.0-193.13.2.el8_2.ppc64le.img
--reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f --add 'ignition' --no-hostonly --add-drivers 'mptspi vmw_pvscsi' --omit-drivers 'nouveau' --omit 'nfs' --add 'iscsi' --add 'ifcfg' --add 'fips' --kver '4.18.0-193.13.2.el8_2.ppc64le'
- With the kernel image, modules, and initrd image available, use the kexec command to load the custom kernel and reboot into it using the kexec command.
# ./kexec -l --append="`cat /proc/cmdline`" --initrd=initramfs-<kver>.img vmlinux-<kver>
# ./kexec -e
In conclusion, use the container for certain tasks, such as installing development packages, compiling a test kernel, or getting binaries for packages (for example, kexec) that are not available on the RHCOS host. Note that these steps are hacky in nature and may need a bit of tweaking over time with security enhancements. So, take this how-to with a pinch of salt. Good luck with your debugging journey!
Things to remember
After reaching initrd-switch-root.target in the systemd boot process  , the modprobe command would fail to find modules as the modules for the custom kernel could not be placed in the default /lib/modules directory (as it is read-only) on the root disk.
You can use the following three options as workaround:
- Use alias modprobe -d $DIRPATH for modprobe.
- Build the missing kernel modules into the initrd with the add-drivers
- Compile the modules into the kernel (use =y instead of =m)
Contacting the Enterprise Linux on Power Team
Have questions for the Enterprise Linux on Power team or want to learn more? Follow our discussion group on IBM Community Discussions.