Infrastructure as a Service

 View Only

KMS Managed Encryption of Root Partitions with IBM Cloud Bare Metal Server on VPC

By Samuel Matzek posted Thu February 29, 2024 03:52 PM

  

Overview

This tutorial provides step-by-step instructions on how to create an encrypted root partition on an IBM Cloud Bare Metal Server on VPC and how to protect the Linux Unified Key Setup (LUKS) encryption keys from being compromised by using either IBM Cloud Hyper Protect Crypto Services (HPCS) or IBM Key Protect as the single point of control to enable or disable access to data across the enterprise. 

Sandeep Batta and George Wilson wrote a tutorial to describing how to use the hpcs-for-luks package to protect LUKS encryption keys for non-root partitions. The hpcs-for-luks package has now been updated to support root/operating partition partitions as well and this tutorial provides those steps.

NOTE: This process has been tested with Red Hat Enterprise Linux 9.2 which contains dracut version 057. Earlier versions of dracut may not work with hpcs-for-luks.

Background

Boot disks contain multiple partitions, and by default boot disks of IBM Cloud Bare Metal Servers on VPC contain 4 partitions: a bios boot partition, an EFI system partition, a Linux extended boot partition which is where the kernels and the initial RAM disk (file systems) are stored, and the root partition which contains the operating system install.

This tutorial describes how to setup an additional partition which will be encrypted with Linux Unified Key Setup (LUKS). The operating system will be loaded into this encrypted partition and the LUKS passphrase used to open the encrypted partition during boot will be wrapped by a key from an IBM Cloud key management system. This allows a single point of control to enable or disable access to the operating systems across the enterprise.

The resulting boot disk partition layout and boot flow operating as follows:

  1. When the server boots the kernel and initial RAM file systems (initramfs) are loaded into RAM from the boot disk.
  2. The hpcs-for-luks service in the initramfs authenticates with IBM Cloud Identity Access Management and receives an authentication token.
  3. The authentication token is used call an API on an IBM key management system to unwrap/decrypt the LUKS passphrase.
  4. The encrypted operating system partition is opened using the passphrase.
  5. The boot process proceeds as normal, mounting the root file system and loading the operating system in it.

Create the bare metal server

The first step is to create and setup the bare metal server in IBM Cloud. On the provisioning screen user data must be specified to keep the stock image operating system in a small partition and leave unpartitioned free space on the boot drive for additional partitions.

The user data value to specify is:

#cloud-config
growpart: 
  mode: off
  devices: ['/']
resize_rootfs: false

The user data field is near the bottom of the provisioning screen:


Configure a file system in an encrypted partition

After the server is provisioned create a new partition, encrypt it with LUKS, and create a file system in the partition.

First, use fdisk to create a new partition. In this example, the root drive is /dev/sda and fdisk is used to add a new 50GiB partition:

# fdisk /dev/sda
This disk is currently in use - repartitioning is probably a bad idea. 
It's recommended to umount all file systems, and swapoff all swap 
partitions on this disk.
Command (m for help): n 
Partition number (6-128, default 6): 
First sector (20971487-1875253854, default 20971520): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (20971520-1875249151, default 1875249151): +50G

Created a new partition 6 of type 'Linux filesystem' and of size 50 GiB

Command (m for help): w 
The partition table has been altered. 
Syncing disks.

The lsblk -f command show the server’s block devices and their partitions:

# lsblk -f
NAME     FSTYPE      FSVER            LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
├─sda2   vfat        FAT16                   7B77-95E7                             192.8M     3% /boot/efi
├─sda3   xfs                          boot   e78d6a6f-15cc-469b-83ab-9a60b7b574f0  226.1M    54% /boot
├─sda4   xfs                          root   d370e124-ea83-46ea-a7ef-67f12dd8bb3c    7.8G    16% /
├─sda5   iso9660     Joliet Extension cidata 2023-10-17-15-35-27-00
└─sda6

Install the cryptsetup package:

dnf install cryptsetup

Encrypt (luksFormat) the /dev/sda6 partition. The luksFormat command will prompt for a passphrase. In the following example the --force-password is specified to allow a trivial passphrase of “MyPhrase” for the purposes of this walkthrough.

cryptsetup luksFormat --type luks2 --force-password /dev/sda6

Next open the encrypted partition. This command will prompt for the passphrase, open the encrypted partition, and create a /dev/mapper/root device for I/O to the partition.

cryptsetup open /dev/sda6 root

Next create a file system in the partition. The following command uses the xfs file system but other file systems could be used:

mkfs.xfs /dev/mapper/root

Install hpcs-for-luks prerequisites

Now that the encrypted partition is created, the hpcs-for-luks package can be installed and configured. This is done on the existing operating system partition so that both the original unencrypted rescue partition and the encrypted partition will have the hpcs-for-luks configuration.


Install the hpcs-for-luks prerequisite packages:


dnf install python3 python3-cryptography python3-pip
pip3 install keyprotect

Obtain the hpcs-for-luks package and install it

Now it's time to locate, download, check, and install the hpcs-for-luks package. This package is provided by IBM and hosted at https://github.com/IBM/hyperprotect/tree/main/hpcs-for-luks

Download the hpcs-for-luks package and SHA256SUMS files from https://github.com/IBM/hyperprotect/tree/main/hpcs-for-luks. You can use the SHA-256 hash of the RPM to ensure that it matches the GitHub repo.

The 2.1-1 RPM is required for the encrypted root partition support:

rpm -i hpcs-for-luks-2.1-1.el9.noarch.rpm

Configure hpcs-for-luks

The hpcs-for-luks utility must be configured to communicate with your KMS. Collect the following configuration information from the Overview tab for your instance on the IBM Cloud portal:

  • Instance-ID
  • Key Management endpoint URL
  • Region-ID

You can gather your Hyper Protect Crypto Service endpoint and instance ID by following the steps in the image:

You gather your Key Protect service endpoint by using the Endpoints tab. You can find your instance ID inside your instance by opening the Details side panel.

Note that while the private endpoints are used in this example, public endpoints for IAM, Key Protect, and HPCS can also be used.

  • Create an IBM Cloud IAM API key to access your resource.
  • A skeleton /etc/hpcs-for-luks.ini file is included in the hpcs-for-luks package. Edit it with the information obtained above.
  • Put "placeholder" for default_crk_uuid for now (it will be set later):

[KP]

api_key = AB0CdEfGHijKlMN--12OPqRStuv3wx456yZAb7CDEF8g

region = us-east

service_instance_id = 01234567-89ab-cdef-0123-456789abcdef

endpoint_url = https://api.private.us-east.hs-crypto.cloud.ibm.com:9730

default_crk_uuid = fedcba98-7654-3210-fedc-ba9876543210

Test the configuration by attempting to list keys in the KMS:

hpcs-for-luks list

If the test fails, check for typos, check on the status of your instance via the portal, diagnose the problem, and retry.

Generate a customer root key (CRK)

Now that the hpcs-for-luks utility is configured, you can use it to generate a CRK:

Generate a CRK in the KMS:

hpcs-for-luks create --crk --gen --name MyCRKName

List the keys to get the ID associated with MyCRKName:

hpcs-for-luks list | grep MyCRKName

Because the CRK is frequently used and seldom changed, it is convenient to configure the hpcs-for-luks utility to use it. Edit /etc/hpcs-for-luks.ini and change the "placeholder" value for default_crk_uuid to the ID you just obtained above:

[KP]

api_key = AB0CdEfGHijKlMN--12OPqRStuv3wx456yZAb7CDEF8g

region = us-east

service_instance_id = 01234567-89ab-cdef-0123-456789abcdef

endpoint_url = https://api.private.us-east.hs-crypto.cloud.ibm.com:9730

default_crk_uuid = fedcba98-7654-3210-fedc-ba9876543210

Wrap the LUKS passphrase

Now that hpcs-for-luks is configured with a root key wrap the passphrase used for the new encrypted root partition. The following command wraps the “MyPhrase” passphrase and stores the wrapped passphrase into the “luks:root” file. The hpcs-for-luks module will use this wrapped phrase to open the encrypted root partition during boot.

echo -n 'MyPhrase' | hpcs-for-luks wrap > /var/lib/hpcs-for-luks/user/luks:root

Add a token to the cryptsetup of the encrypted partition. This updates the configuration of the encrypted partition to use a Linux kernel key for opening the partition.

cryptsetup token add /dev/sda6 --key-description luks:root

Enable the hpcs-for-luks-wipe service which removes the keys hpcs-for-luks adds to the kernel key ring when the boot process is finished with them.

systemctl enable hpcs-for-luks-wipe

Test hpcs-for-luks processing

Test the hpcs-for-luks processing of the wrapped passphrase and the automatic opening of the partition.

First, close the encrypted partition:

cryptsetup close root

Next, have hpcs-for-luks process the wrapped key and add it to the Linux kernel keyring:

hpcs-for-luks process

Open the encrypted partition. This command should run and open the partition without asking for a passphrase. If it prompts for a passphrase there is a configuration problem from one of the previous steps which needs to be fixed before proceeding.

cryptsetup open /dev/sda6 root

Configure dracut modules and rebuild the initramfs

Now that hpcs-for-luks is configured to open the encrypted root partition with the wrapped passphrase, configure the required dracut modules for hpcs-for-luks and rebuild the initramfs to include the hpcs-for-luks module.

This builds the wrapped passphrase file into the initramfs and configures hpcs-for-luks in the initial boot steps so it can unwrap and add the LUKS passphrase to a kernel key so LUKS can automatically open the encrypted root partition.

Add the required modules to the /etc/dracut.conf.d/hpcs-for-luks.conf file so that the contents of the file look like this:


add_dracutmodules+=" hpcs-for-luks crypt ifcfg network network-manager url-lib"

The initramfs can now be rebuilt to include the wrapped passphrase and the hpcs-for-luks module.

Dracut version 057 which is included in Red Hat Enterprise Linux 9.2 has a bug which assumes that dracut is being run with a umask value of 022. This is not the default umask of the root user in the stock image. Therefore, the umask must be changed to 022 before running dracut and set back afterward. The following commands change umask around the dracut command to correctly rebuild the initramfs:

INITIAL_UMASK=`umask`
umask 022
dracut --force --verbose
umask $INITIAL_UMASK

Increase the GRUB menu timeout value

When the bare metal server boots a menu is shown on the console which allows the user to select different boot configurations. Later in this tutorial a new boot menu option will be added for the encrypted root partition and the encrypted root partition option will be set as the default.


The default timeout value to choose a boot configuration is 2 seconds. If you want to change this value to give more time to select the non-encrypted operating system partition it is best to do it now before the operating system is copied into the encrypted partition.

1.         Edit /etc/default/grub
2.         Change the GRUB_TIMEOUT value to your desired time in seconds.
3.         Run the following two commands:


grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg

Copy the unencrypted root partition operating system to the encrypted file system

Now that the hpcs-for-luks configuration is complete the next step is to copy the operating system into the encrypted partition and file system.

First mount the encrypted root file system:

mkdir /mnt/encryptedroot
mount /dev/mapper/root /mnt/encryptedroot

 

Next copy the operating system into the encrypted root:


rsync -a --exclude='/proc/*' --exclude='/sys/*' --exclude='/boot' --exclude='/mnt/encryptedroot' / /mnt/encryptedroot

After the operating system is copied, the update fstab and crypttab files for the encrypted root partition.

The crypttab file need values from the lsblk output:

lsblk -f
NAME     FSTYPE      FSVER            LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
├─sda2   vfat        FAT16                   7B77-95E7                             192.8M     3% /boot/efi
├─sda3   xfs                          boot   e78d6a6f-15cc-469b-83ab-9a60b7b574f0  226.1M    54% /boot
├─sda4   xfs                          root   d370e124-ea83-46ea-a7ef-67f12dd8bb3c    7.7G    17% /
├─sda5   iso9660     Joliet Extension cidata 2023-11-30-13-54-10-00                             
└─sda6   crypto_LUKS 2                       8a374993-d8f4-42ff-92ce-25da7977b7c0                 └─root                                                                                        

Edit or create the /mnt/encryptedroot/etc/crypttab file and add this line:

root UUID=8a374993-d8f4-42ff-92ce-25da7977b7c0 none luks,_netdev

The UUID value should come from the crypto_LUKS 2 partition that was added for the encrypted root.

Edit the /mnt/encryptedroot/etc/fstab file. Remove the line that has the single “/” for the root file system and add following line in its place:

/dev/mapper/root   /       xfs     defaults        0       1

Add the encrypted root boot as GRUB option

To boot from the encrypted root partition, it needs to be added as an option in GRUB. To add this option, first look at the existing entries using grubby command:

grubby --info ALL

Using the same kernel and initrd name as the index=0 value from the above command, add the encrypted root partition option with the following command. Note the rd.luks.name value is the UUID of the sda6 crypto_LUKS2 partition from the lsblk -f output above.

grubby --add-kernel="/boot/vmlinuz-5.14.0-284.25.1.el9_2.x86_64" \
--title="Boot from encrypted root" \
--initrd="/boot/initramfs-5.14.0-284.25.1.el9_2.x86_64.img" \
--args="ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 rd.shell rd.hpcs-for-luks log_buf_len=1M rd.neednet=1 root=/dev/mapper/root rd.luks.name=8a374993-d8f4-42ff-92ce-25da7977b7c0=root ip=dhcp"

Verify the kernel/menu option was added and get its index number.


grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-284.25.1.el9_2.x86_64"
args="ro no_timer_check net.ifnames=0 console=ttyS1,115200n8 console=tty0 efi=noruntime"
root="UUID=d370e124-ea83-46ea-a7ef-67f12dd8bb3c"
initrd="/boot/initramfs-5.14.0-284.25.1.el9_2.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-284.25.1.el9_2.x86_64) 9.2 (Plow)"
id="5ab34399c941d21ae071deafd950883c-5.14.0-284.25.1.el9_2.x86_64"
index=1
kernel="/boot/vmlinuz-5.14.0-284.25.1.el9_2.x86_64"
args="ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 rd.shell rd.hpcs-for-luks log_buf_len=1M rd.neednet=1 ip=dhcp rd.luks.name=8a374993-d8f4-42ff-92ce-25da7977b7c0=root"
root="/dev/mapper/root"
initrd="/boot/initramfs-5.14.0-284.25.1.el9_2.x86_64.img"
title="Boot from encrypted root"

In the above output the new boot option has been added with index=1. To make the bare metal server boot from the encrypted partition by default, set the default index to 1 using the following command:

grubby --set-default 1

With IBM Cloud Bare Metal Servers on VPC and Red Hat Enterprise Linux 9.2, the grubenv file must be copied for the default index change to take effect:

cp /boot/grub2/grubenv /boot/efi/EFI/redhat/

The system is now ready to reboot and boot from the encrypted root partition.

Rebooted server view

Upon reboot the system console will show the new GRUB menu entry as shown in this screen shot:

The system should boot fully without asking for the LUKS passphrase for the root partition on the system console.

After reboot the lsblk -f command shows that the root file system is mounted from the encrypted root partition:

lsblk -f

NAME     FSTYPE      FSVER            LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS

sda
├─sda1                                                                                          
├─sda2   vfat        FAT16                   7B77-95E7                             192.8M     3% /boot/efi
├─sda3   xfs                          boot   e78d6a6f-15cc-469b-83ab-9a60b7b574f0  225.7M    54% /boot
├─sda4   xfs                          root   d370e124-ea83-46ea-a7ef-67f12dd8bb3c               
├─sda5   iso9660     Joliet Extension cidata 2023-11-30-13-54-10-00                             
└─sda6   crypto_LUKS 2                       8a374993-d8f4-42ff-92ce-25da7977b7c0               
  └─root xfs                                 255e3ce7-5510-417e-892e-bac89a408828     48G     4% /

Conclusion


Using the hpcs-for-luks package you can encrypt the root operating system partitions of your bare metal servers, encrypt the LUKS passphrase, and centrally manage the access to decryption of the passphrase using an IBM Cloud key management system.

0 comments
17 views

Permalink