Configure 16 GB HugePage on IBM PowerVM systems running Linux
Aneesh Kumar(aneesh.kumar@in.ibm.com), Vaibhav Jain(vajain21@in.ibm.com),
Geetika Moolchandani(Geetika Moolchandani1@ibm.com)
This document details the steps to configure 16GB HugePages on an IBM® PowerVM® system running the Linux® operating system in hash memory management unit (MMU).
There are two major set up tasks that need to be performed before a Linux partition can make use of 16 GB HugePages in the operating system.
First, configure the HugePage pool at the PowerVMsystem level and then configure the number of HugePages at the Linux partition level.
Procedure to configure HugePages on PowerVM
-
Prepare the system
-
Set up HugePage memory pool
-
Configure HugePages in the Linux partition profile
-
Configure HugePages at the OS level
-
Verify the configured HugePages
-
Run the libhugetlb tests to verify 16GB Hugeage access
-
Use troubleshoot tips for common issues
System preparation
The system environment used were IBM POWER9™ and RHEL/SLES Linux distributions. However, the steps listed are applicable to any PowerVM system (including IBM POWER8®, POWER9, and so on in hash MMU.
Set up HugePage memory pool
Perform the following steps to set up a HugePage memory pool through an Advanced System Management Interface (ASMI), which is a web-based interface of the Power system.
-
Switch off the IBM Power® system.
-
Log in to ASMI web interface by using the default user ID and password (in this example, the user ID is dev and the password is FipSdev, which is the default user ID and password).
-
On the ASMI main page, expand Performance Setup.
- Click System Memory Page Setup and on the corresponding page, specify the number of huge pages to be configured.
In above example, you can see 10 HugeP-ages has been configured.
5. Click Save settings.
6. Power on the system by clicking Save settings and Power on.
Note:
If you have a full resource system without being managed by Hardware Management Console (HMC), that is, a non-logical partition (LPAR)-based system, you can skip the Configure HugePage in the Linux partition step and continue with the Configure HugePages at the OS level step.
Configure HugePage in the Linux partition
-
Log in to the HMC where the managed system is hosted.
-
Select the Linux partition that needs to be configured with HugePages.
-
To edit the partition profile, and click the Memory tab.
There are 20 HugePages can be configurable for an LPAR.
Specify “Minimum pages” ----> 2
“Desired pages” -----> 4
“Maximum pages” -----> 10
4. Save the changes and activate the partition.
Configure HugePages at the OS level
Use the GRUB edit method to configure HugePage size and the number of huge-pages during boot time. This is a mandatory step to configure HugePages during boot time.
1. Boot the operating system (at the time of this documentation we used RHEL 8).
2. Configure the HugePage size and the number of HugePages as the boot parameter.
Edit the grub file “/etc/default/grub” and specify HugePage size and request for number of pages. Add as part of GRUB_CMDLINE_LINUX.
"hugepagesz=16G hugepages=4"
Ex: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel_XXXX/root rd.lvm.lv=rhel_XXXX/swap biosdevname=0 hugepagesz=16G hugepages=4"
For the above GRUB changes to take effect, update the GRUB using the “grub2” tool as specified below:
[root@XXXX ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Generating boot entries from BLS files...
Done
Reboot the partition to view the updates.
[root@XXXX ~]# reboot
Verify the configured HugePage
At the OS level, there are multiple ways to ensure that HugePages are configured properly.
Use the following methods to check if HugePages are configured correctly on the operating system:
Check the kernel command line to verify the boot parameters with 16GB HugePages.
[root@XXXX ~]# dmesg | grep huge
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.18.0-64.el8.ppc64le root=/dev/mapper/rhel_XXXX-root ro crashkernel=auto rd.lvm.lv=rhel_xxxxx-lp1/root rd.lvm.lv=rhel_XXXX/swap biosdevname=0 hugepagesz=16G hugepages=4
-
If the user has configured the HugePage pool and requested for a number of HugePages using the boot parameter, verify using the dmesg log.
[ 0.000000] Huge page(16GB) memory: addr = 0xF9C00000000 size = 0x400000000 pages = 1
For example, if the HugePage pool is configured with 10 HugePages , then OS will return 10 entries for HugePage memory addr, as specified below:
linux-zzyi:~ # dmesg | grep -i "Huge page"
[ 0.000000] Huge page(16GB) memory: addr = 0xF9C00000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFA000000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFA400000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFA800000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFAC00000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFB000000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFB400000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFB800000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFBC00000000 size = 0x400000000 pages = 1
[ 0.000000] Huge page(16GB) memory: addr = 0xFC000000000 size = 0x400000000 pages = 1
linux-zzyi:~ # dmesg | grep "Huge page" | wc -l
10
## check dmesg for HugeTLB
This indicates 16 GB HugePages are configured with pre-allocated pages as shown below.
linux-zzyi:~ # dmesg | grep HugeTLB
[ 20.429376] HugeTLB registered 16 GB page size, pre-allocated 4 pages
[ 20.429395] HugeTLB registered 16 MB page size, pre-allocated 0 pages
If 16 GB HugePages are configured, check if nr_hugepages is matching with what is pre-allocated. In this example, 4 HugePages are pre-allocated.
linux-zzyi:~ # cat /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages
4
Run libhugetlb tests to verify 16G HugePage access
Mount the configured 16 GB HugePage.
linux-zzyi:~ # mkdir /mnt/16G
linux-zzyi:~ # mount -o pagesize=16G -t hugetlbfs none /mnt/16G
linux-zzyi:~ # mount | grep huge
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
none on /mnt/16G type hugetlbfs (rw,relatime,pagesize=16G)
Build the libhugetlb source code and run the tests.
-
-
git clonehttps://github.com/libhugetlbfs/libhugetlbfs.git
-
make (this will build the source)
[root@XXXX libhugetlbfs]# make
VERSION
version update: 2.21
version string: 2.21
CC64 obj64/elflink.o
AS64 obj64/sys-elf64lppc.o
CC64 obj64/elf64lppc.o
CC64 obj64/hugeutils.o
CC64 obj64/version.o
version.c:3:19: warning: ‘libhugetlbfs_version’ defined but not used [-Wunused-const-variable=]
static const char libhugetlbfs_version[] = "VERSION: "VERSION;
^~~~~~~~~~~~~~~~~~~~
CC64 obj64/init.o
CC64 obj64/morecore.o
CC64 obj64/debug.o
....
....
Enter the make check command to run the tests.
[root@XXXXX libhugetlbfs]# make check
zero_filesize_segment (16G: 64): PASS
test_root (16G: 64): PASS
meminfo_nohuge (16G: 64): PASS
gethugepagesize (16G: 64): PASS
gethugepagesizes (16G: 64): PASS
HUGETLB_VERBOSE=1 empty_mounts (16G: 64): PASS
HUGETLB_VERBOSE=1 large_mounts (16G: 64): PASS
find_path (16G: 64): PASS
unlinked_fd (16G: 64): PASS
readback (16G: 64): PASS
truncate (16G: 64): PASS
shared (16G: 64): PASS
mprotect (16G: 64): PASS
mlock (16G: 64): PASS
misalign (16G: 64): PASS
fallocate_basic.sh (16G: 64): PASS
fallocate_align.sh (16G: 64): PASS
ptrace-write-hugepage (16G: 64): PASS
icache-hygiene (16G: 64): PASS
....
....
Trouble shooting tips
Scenario 1:
You have not configured the HugePage pool, but passed the kernel boot parameter to get HugePages.
For example, passed hugepagesz=16G hugepages=4 as the boot parameter. This will result in not pre-allocating HugePages at boot time.
Verify this from the dmesg log.
No HugePage pool is configured. So, the OS will return 0.
linux-zzyi:~ # dmesg | grep "Huge page" | wc -l
0
Though HugePages are requested during boot, the pool is not configured, and therefore, pre-allocated pages will be 0.
linux-zzyi:~ # dmesg | grep HugeTLB
[ 4.210982] HugeTLB registered 16 GB page size, pre-allocated 0 pages
[ 4.210987] HugeTLB registered 16 MB page size, pre-allocated 0 pages
Scenario 2:
16 GB HugePages are always pre-allocated during boot time. If it is not pre-allocated and you try to echo value to nr_hugepages” , the user will encounter the error “...write error: Invalid argument”.
linux-zzyi:~ # echo 1 > /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages
-bash: echo: write error: Invalid argument
Summary
This document explained how to configure and run libhugetlb tests on PowerVM systems running Linux operating System with hash MMU. Additionally, it describes two common scenarios the users might come across while configuring HugePages and this helps in troubleshooting the issue and getting it resolved.