Authors: Pavithra Prakash (pavrampu@in.ibm.com) , Donet Tom (Donet.Tom@ibm.com)

1. Abstract

This document provides a detailed guide on configuring HugePages on IBM Power Systems (P9, P10, and P11) running Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES). It covers:

- Supported HugePage sizes across different Power architectures and Memory Management Unit [MMU] modes (Hash vs. Radix)

- Step-by-step instructions for configuring 2MB, 1GB, 16MB, and 16GB HugePages

- Temporary and persistent configuration methods

- Platform-specific considerations, including firmware-level setup for 16GB HugePages

2. Introduction

As modern applications grow in complexity and memory demands, optimizing memory management becomes critical especially on enterprise grade hardware like IBM Power Systems. One such optimization is the use of HugePages, a Linux kernel feature that allows memory to be managed in larger chunks than the default page size which is 64k. By reducing the number of pages the system must track, HugePages can significantly improve performance for memory-intensive workloads.

Below is the output of a sample program which shows two different scenarios one without and one with hugetlbfs, with hugethbfs the significantly lower number of page faults demonstrates that hugetlbfs is more efficient in handling memory access, as it reduces the need for disk I/O and improves overall performance.

2.1 Supported Hugepage Sizes

3. Configuring 2MB Hugepages (Radix Mode Default)

To configure 2MB hugepages temporarily echo the number of hugepages to /proc/sys/vm/nr_hugepages file.

echo <nr_hugepages> > /proc/sys/vm/nr_hugepages

or

echo <nr_hugepages> > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

To verify the configuration:

cat /proc/sys/vm/nr_hugepages

tail /proc/meminfo

Ex:

ltcden14-lp5:~ # echo 200 > /proc/sys/vm/nr_hugepages

ltcden14-lp5:~ # cat /proc/sys/vm/nr_hugepages

200

ltcden14-lp5:~ # tail /proc/meminfo

HugePages_Total: 200

HugePages_Free: 200

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

Hugetlb: 409600 kB

DirectMap4k: 0 kB

DirectMap64k: 0 kB

DirectMap2M: 10551296 kB

DirectMap1G: 1075838976 kB

To make hugepage configuration persistent across reboots we can add the hugepage kernel command line parameters.

RHEL:

We can use grubby tool to update the kernel command line parameters or add it in /etc/default/grub.

In the steps below considering 160 as example for number of hugepages.

Using grubby tool:

1. grubby --args="hugepagesz=2MB hugepages=160" --update-kernel=/boot/vmlinuz-$(uname -r)

2. reboot

Editing grub file:

1. vi /etc/default/grub

2. Add “hugepagesz=2MB hugepages=160” to GRUB_CMDLINE_LINUX

3. Run “grub2-mkconfig -o /boot/grub2/grub.cfg”

4. reboot

SLES:

We can add "hugepagesz=2MB hugepages=160" in /etc/default/grub to update kernel command line parameters. In SLES15 releases we can even use yast2 tool to add kernel command line parameters

using yast2:

- Run yast2

- Navigate to System → Boot Loader

- Under “Kernel Parameters” Add “hugepagesz=2MB hugepages=160” to Optional Kernel Command Line Parameter

- Select OK

- Reboot the system

Editing grub file:

1. vi /etc/default/grub

2. Add “hugepagesz=2MB hugepages=160” to GRUB_CMDLINE_LINUX_DEFAULT

3. Run “grub2-mkconfig -o /boot/grub2/grub.cfg”

4. reboot

Ex:

# cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinux-6.16.0-rc1-160000.11-default root=UUID=20e98e87-7a45-4279-aa79-7eedccf02176 BOOT_IMAGE=ceca22a89502/boot/ppc64le/linux rd.neednet=1 quiet security=selinux selinux=1 enforcing=1 hugepagesz=2MB hugepages=160

ltcfuji8V4-lp3:~ # tail /proc/meminfo

HugePages_Total: 160

HugePages_Free: 96

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

Hugetlb: 327680 kB

DirectMap4k: 0 kB

DirectMap64k: 0 kB

DirectMap2M: 7995392 kB

DirectMap1G: 425721856 kB

ltcfuji8V4-lp3:~ #

4. Configuring 1GB Hugepages

Temporary:

echo <nr_hugepages> > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Ex:

echo 20 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Persistent (across reboots):

To make hugepage configuration persistent across reboots we can add the hugepage kernel command line parameters using the same steps as explained in 2MB section [ Section 3 ], But the kernel commandline parameters will be as below.

Passing “ hugepagesz=1GB hugepages=11” configures 11 1GB hugepages but default hugepage size will be still 2MB

Ex:

localhost:~ # cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinux-6.4.0-150600.23.47-default root=UUID=d180e3b4-0a33-4555-b723-48d4fd6cc5a0 splash=verbose quiet security=apparmor crashkernel=490M hugepagesz=1GB hugepages=11 mitigations=auto

localhost:~ # tail /proc/meminfo

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

Hugetlb: 11534336 kB

DirectMap4k: 0 kB

DirectMap64k: 0 kB

DirectMap2M: 1048576 kB

DirectMap1G: 208666624 kB

localhost:~ # cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

11

To make 1GB hugepage as default hugepage pass “ default_hugepagesz=1GB hugepagesz=1GB hugepages=<nr_hugeapges>”

Ex:

grubby --args="default_hugepagesz=1GB hugepagesz=1GB hugepages=800" --update-kernel=/boot/vmlinuz-$(uname -r)

5. Configuring 16MB Hugepages (Default Hash Mode)

In Hash mode 16MB is the default hugepage size.

Note: In P9 and below default is Hash mode on P10 and P11 we can change the mode to Hash by passing disable_radix [ refer Section 7 ]

file: /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages

kernel parameters: default_hugepagesz=16MB hugepagesz=16MB hugepages=16

We can use the same steps mentioned in 2MB configuration section to update the kernel parameters.

Ex:

localhost:~ # echo 100 > /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages

localhost:~ # cat /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages

100

localhost:~ # tail /proc/meminfo

FileHugePages: 0 kB

FilePmdMapped: 0 kB

CmaTotal: 0 kB

CmaFree: 0 kB

HugePages_Total: 100

HugePages_Free: 100

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 16384 kB

Hugetlb: 1638400 kB

localhost:~ #

grubby --args="default_hugepagesz=16MB hugepagesz=16MB hugepages=16" --update-kernel=/boot/vmlinuz-$(uname -r)

6. Configuring 16GB Hugepages

16GB hugepages are hardware-supported

Unlike smaller hugepage sizes (like 2MB or 1GB) which are purely software-managed by the Linux kernel, 16GB hugepages require configuration at the firmware level, specifically through BMC or FSP.

This setup ensures that the system reserves physical memory blocks of 16GB during boot, making them available to the OS for use as hugepages.

6.1 How to configure:

1. For configuring 16GB hugepages on lpar first we need to configure 16GB hugepages in BMC/FSP.

FSP:

- Power off the entire system

- Login to ASM page

- Navigate to Performance Setup → System Memory Page Setup

- Configure the number of 16GB hugepages and power on the system

BMC:

- Power off the system

- Login to ASMI page

- Navigate to “Resource Management” → Memory → System memory page setup

- Configure number of hugepages in “Requested huge page memory”

- Click “update system memory page setup”

2. Update HMC LPAR profile with number of 16GB hugepages

- Shutdown the lpar

- Navigate to profile → manage profile → Memory

- Under “Huge Page Memory” Configure the number of 16GB hugepages required [ should be less than “configurable pages” ]

- Save Profile

- Activate the lpar

3. Lastly configure the number of 16GB hugepages from LPAR command line.

Parameters: default_hugepagesz=16GB hugepagesz=16GB hugepages=4

File: /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages

Note: Pass “disable_radix” if it is Radix system

7. Changing MMU from Radix to Hash mode

On Radix system we can change MMU from Radix to Hash mode by passing “disable_radix” kernel command line argument. We can add the kernel command line argument using grubby tool or by editing /etc/default/grub file as explained in “Configuring 2MB hugepages” section

We can verify the MMU mode in /proc/cpuinfo file

Ex:

[root@ltcblue8vb-lp12 ~]# cat /proc/cpuinfo | grep MMU

MMU : Hash

8. Conclusion

Hugepages are a powerful performance optimization tool in Linux, especially on IBM Power systems (P9, P10, P11). By allowing memory to be managed in larger chunks, hugepages reduce the overhead of page table management and TLB [ Translation Lookaside Buffer ] misses both of which are critical for memory-intensive workloads.

9. Acknowledgment

We would like to extend our sincere thanks to Manu, Sachin Sant, Ritesh and Pavithra V for taking time to review this blog. Your insights and feedback have been incredibly valuable in refining the content and ensuring its accuracy.