Loading a KTAP during STAP installation

Author : Shashank Pinjarkar

Introduction

The blog revolves around KTAP - a kernel level module working with STAP to intercept traffic. Its loading during STAP installation depends primarily on your server's running kernel.

Each Linux type (Red Hat, SUSE, Ubuntu, to name a few) keeps releasing new kernels whenever there is a new release. For example, recently Red Hat 8.10 was released and that brought with it a new family of kernels. Certain OS like Amazon Linux , it’s releasing kernels at 3x the rate compared to Red Hat. Not every single kernel will have a matching KTAP module. Does that mean KTAP can’t be loaded? Read the blog to find out 

Prerequisites

Linux - running a shell script, installing new packages, brief knowledge about kernels and how to change default kernel

Knowledge about Unix-STAP (how to install should be more than enough)

There are multiple ways of installing STAP but all of them just fall under two categories - Shell (Shell installation, Unified installation, Native/RPM installation) and via GIM (GIM installation, consolidated GIM installation)

I will focus on ‘Shell’ category as it is easy to explain. Moreover, the terminal displays the output of what’s happening behind the scenes. The images I have attached are from during STAP installation via Shell. In case, you have STAP installation via GIM, you can view the KTAP logs (located by default at /var/log/ktap_install.log)

Note: KTAP modules list is a tar file (.tgz) that comes with the zip file when you download STAP from IBM Fix Central . It contains a list of kernel object (.ko) files.

Some helpful commands -

1. To check which kernel your server is using - uname -r
2. To check if KTAP is loaded or not (works on Linux only) - lsmod | grep ktap
3. List of packages installed for your kernel -
   1. On Red Hat based systems - rpm -qa | egrep <kernel name>
   2. On Ubuntu based systems - dpkg --list | grep <kernel name>

Type 1:  Exact match of KTAP

This is the simplest case. Your kernel has an exact matching KTAP module present in the module list. All that you need to do is just run STAP installer the default way. No extra flags need to be passed. A KTAP will load easily. When the installation completes, it will show a message like -

Module ktap-xxxxxxxxxxxxxxxxxxxxxxxx-4.18.0-425.19.2.el8_7.x86_64-x86_64-SMP.ko selected for kernel 4.18.0-425.19.2.el8_7.x86_64.
Extracted module ktap-xxxxxxxxxxxxxxxxxxxxxxxx-4.18.0-425.19.2.el8_7.x86_64-x86_64-SMP.ko from /usr/local/guardium/guard_stap/ktap/current/modules-xxxxxxxxxxxxxxxxxxx.tgz

Here is a sample output of above case -

Type 2: Flex Load

Now, there are hundreds if not thousands of kernels across all Linux platforms. Not all of them are part of our STAP’s module list. For example, there may not be an exact match for kernel-4.18.0-348.23.1.el8_5.x86_64. 

Does that mean a KTAP can’t be loaded? No, we are yet to reach that state. Even though there is no exact match, there may be another KTAP module that closely matches your running kernel. When I say ‘closely matches’, it means they are part of same kernel family. Flex load means a closely matching or loose fitting (to be precise, the best fit) KTAP module with your running kernel will be selected during STAP installation.

More about kernel family:

Generally, a kernel version looks like this: X.Y.Z-A.B.C. The first triplet (X.Y.Z) represents the primary kernel version and defines the kernel family. Kernels with the same X.Y.Z belong to the same kernel family, provided the operating system (OS) and architecture are also the same. For example, using native package manager you can't load a SUSE kernel on a Red Hat system or use an x86 kernel on an ARM-based machine.

For flex load mechanism to work, here are the conditions to look for -

• your running kernel’s family should be supported by STAP - meaning there must be at least one KTAP module in module list whose X.Y.Z matches with your kernel.
• digits B.C of your kernel must be higher compared to the one in module list. For example, if your running kernel is 4.18.0-193.28.1.el8_2.x86_64 and the closest lower kernel in module list is 4.18.0-193.el8.x86_64-x86_64, then flex load can potentially work. 
• In all cases, the kernel module representing the most recent kernel that matches the rules, and also is older than the kernel version installed, is chosen. Module families also need to match, for example, el5, el6, el7, pae, x86_64, and so on.

Now, flex load is of two types - Implicit or Explicit

Implicit Flex Load

Apart from the KTAP module list, there is one more list. It’s called ktaposmatch.csv. It comes along with the zip file when you download STAP from IBM Fix Central. This file contains two columns - 

• Column A - Supported OS kernel level (uname -r)
  
• Column B - KTAP module 

Sample ktaposmatch.csv file

To check if implicit flex load with work or not, you need to check if your running kernel is in Column A or not. If yes, then the corresponding entry in Column B will be selected as the best fit KTAP module during installation. The mechanism is called implicit because you don’t need to pass any extra flags during STAP installation. The best fit KTAP module will be implicitly selected. A sample output of such a case is -

Explicit Flex Load

In case your kernel entry is not present in Column A but it meets the flex load criteria, then there is possibility that KTAP can be loaded. You just need to explicitly mention that during STAP installation. Just pass the flag --ktap_allow_module_combos (In case if you have STAP installed via GIM, just reinstall/update STAP with KTAP_ALLOW_MODULE_COMBOS=Y).

Here is sample output of Explicit Flex Load -

What if you don’t pass the above flag? Well, STAP will get installed but without any KTAP. At the end of installation, you will see a message as shown in below image -

Follow the steps in red box, and KTAP will load successfully.

Type 3: Custom Build

What difference do you see in the success output of Implicit and Explicit flex load? If you take a close look, you will observe that before going for a best fit option, the Explicit Flex Load says a “Local build of KTAP will not be attempted.  Please install kernel development packages for ….”

A custom build or local build is when a new KTAP module is built on the fly, during the STAP installation. It just requires the necessary kernel development packages to be pre-installed for your running kernel.  The packages vary across Linux platforms. Here is a sample list -

• Red Hat

• • kernel-core
  • kernel-modules
  • kernel-devel
  • kernel-headers
  • gcc

• SUSE

• • kernel-default
  • kernel-default-devel
  • kernel-devel
  • kernel-source
  • gcc, make, binutils

•  Ubuntu / Debian

• • linux-image
  • linux-headers
  • build-essential
  • gcc

Note: You need to make sure you are installing the packages for your booted kernel

Sample output of a custom build getting generated for a non-supported kernel -

A lot happens during a custom compilation as it is building a KTAP from scratch. And thus, it has its own limitations. If the kernel family for your running kernel is not supported by STAP’s module list, then the chance of a successful custom build decreases significantly.

In case, all ways of loading a KTAP fails for you, or STAP does not get installed at all due to non-supporting kernel family, then the only option is to request a new KTAP to IBM. Once request is full filled, you can download the installers from IBM Fix Central

Thank you for taking the time to read this blog. I hope you found it insightful and worth your while. I would greatly appreciate your feedback, suggestions, or constructive criticism to help me improve and grow.

If you would like to know more about KTAP, I have attached some links from IBM documentation. Feel free to explore them -

Link 1 : https://www.ibm.com/docs/en/gdp/12.x?topic=tap-linux-unix-preparing-install-k

Link 2 : https://www.ibm.com/docs/en/gdp/12.x?topic=tap-linux-unix-s-compilation-k

Link 3 : https://www.ibm.com/docs/en/gdp/12.x?topic=tap-linux-unix-enrolling-k-signing-key

Link 4 : https://www.ibm.com/docs/en/gdp/12.x?topic=luwkt-linux-unix-enable-k-tap-after-installation-if-p-cap-was-installed-by-default

Link 5 : https://www.ibm.com/docs/en/gdp/12.x?topic=tap-linux-unix-requesting-k-module

Author

Shashank Pinjarkar

Unix STAP System Test team

Email: Shashank.Pinjarkar1@ibm.com

Slack: @Sha2nk