Enterprise Linux

Enterprise Linux on Power

Enterprise Linux on Power delivers the foundation for your open source hybrid cloud infrastructure with industry-leading cloud-native deployment options.

View Only

Back to Library

Create persistent memory devices with the Linux device mapper on Power Systems

Thu December 16, 2021 08:10 AM

Kalpana Shetty

Create persistent memory devices with the Linux device mapper on Power Systems - Using persistent memory as a backing storage device on IBM Power10 and Power9 processor-based systems

Contacts:
Aneesh Kumar(aneesh.kumar@in.ibm.com), Vaibhav Jain(vajain21@in.ibm.com),
Geetika Moolchandani(Geetika Moolchandani1@ibm.com)

This document describes how to use the Linux Volume Manager (LVM) to create Linux Device Mapper’s linear logical volumes using persistent memory as the backing storage device on IBM Power10 and Power9 processor-based systems.

What is the purpose of using Linux device mapper with persistent memory?

There are times when applications require volumes larger than what a single dual inline memory module (DIMM), region, or namespace can provide. In such situations, a user can create a larger persistent memory volume by using Linux Device Mapper, which helps to get a single device persistent device or volume.

This document focuses on using a linear device mapper to create LVM volumes on persistent memory devices on Power9 processor-enabled persistent memory systems.

Setup

Power9: Machine type and model 9080-M9S with FW940.27
Hardware Management Console (HMC): V9R1 M940
Linux distribution: SLES 15 SP1 with latest maintweb updates
ndctl version: 64.1
Kernel version: kernel-4.12.14-197.26-default

Preparation

Install SLES 15 SP1 or RHEL 8.2 operating system on the IBM PowerVM® system partition on any Power9 processor-based system.
Apply SLES 15 SP1 maintweb updates.
Install the ndctl tool.

Procedure

Perform the following steps to create the Linux device mapper on persistent memory:

Step 1: Create a logical unit number (LUN) of size 16 TB with affinity option.

Refer to Appendix-A for more details.

Step 2: Activate the partition and make sure that SLES 15 SP1 OS starts successfully.

linux-6ivl:~ # cat /etc/os-release

NAME="SLES"

VERSION="15-SP1"

VERSION_ID="15.1"

PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"

ID="sles"

ID_LIKE="suse"

ANSI_COLOR="0;32"

CPE_NAME="cpe:/o:suse:sles:15:sp1"

linux-6ivl:~ # uname -a

Linux linux-6ivl 4.12.14-197.26-default #1 SMP Wed Nov 6 13:59:49 UTC 2019 (493622b) ppc64le ppc64le ppc64le GNU/Linux

linux-6ivl:~ #

# cat /etc/os-release

NAME="Red Hat Enterprise Linux"

VERSION="8.2 (Ootpa)"

ID="rhel"

ID_LIKE="fedora"

VERSION_ID="8.2"

PLATFORM_ID="platform:el8"

PRETTY_NAME="Red Hat Enterprise Linux 8.2 Beta (Ootpa)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:beta"

HOME_URL="https://www.redhat.com/"

BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"

REDHAT_BUGZILLA_PRODUCT_VERSION=8.2

REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"

REDHAT_SUPPORT_PRODUCT_VERSION="8.2 Beta"

# uname -a

Linux aaa.bbb.ccc.ddd.com 4.18.0-167.el8.ppc64le #1 SMP Sun Dec 15 01:20:45 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

Step 3: Check for regions created by the OS for volume size of 16 TB

From the following output, note that the OS has created eight regions (staring from region0 to region7) for the 16 TB volume size.

linux-6ivl:~ # ndctl list -Ru

[

{

"dev":"region4",

"size":"1638.00 GiB (1758.79 GB)",

"available_size":"1638.00 GiB (1758.79 GB)",

"max_available_extent":"1638.00 GiB (1758.79 GB)",

"type":"pmem",

"iset_id":"0x43f28dca85f9ea6d",

"persistence_domain":"unknown"

{

"dev":"region6",

"size":"2457.75 GiB (2638.99 GB)",

"available_size":"2457.75 GiB (2638.99 GB)",

"max_available_extent":"2457.75 GiB (2638.99 GB)",

"type":"pmem",

"iset_id":"0xcc9372a17e6abb7",

"persistence_domain":"unknown"

{

"dev":"region1",

"size":"1705.25 GiB (1831.00 GB)",

"available_size":"1705.25 GiB (1831.00 GB)",

"max_available_extent":"1705.25 GiB (1831.00 GB)",

"type":"pmem",

"iset_id":"0x48be51b9305fbd4d",

"persistence_domain":"unknown"

{

"dev":"region3",

"size":"2726.50 GiB (2927.56 GB)",

"available_size":"2726.50 GiB (2927.56 GB)",

"max_available_extent":"2726.50 GiB (2927.56 GB)",

"type":"pmem",

"iset_id":"0x21f946e5c27cf536",

"persistence_domain":"unknown"

{

"dev":"region5",

"size":"2457.50 GiB (2638.72 GB)",

"available_size":"2457.50 GiB (2638.72 GB)",

"max_available_extent":"2457.50 GiB (2638.72 GB)",

"type":"pmem",

"iset_id":"0x86e41b950bf3d5db",

"persistence_domain":"unknown"

{

"dev":"region7",

"size":"1639.75 GiB (1760.67 GB)",

"available_size":"1639.75 GiB (1760.67 GB)",

"max_available_extent":"1639.75 GiB (1760.67 GB)",

"type":"pmem",

"iset_id":"0x19926f542ecc576f",

"persistence_domain":"unknown"

{

"dev":"region0",

"size":"1281.00 GiB (1375.46 GB)",

"available_size":"1281.00 GiB (1375.46 GB)",

"max_available_extent":"1281.00 GiB (1375.46 GB)",

"type":"pmem",

"iset_id":"0x24dfa85c98afde26",

"persistence_domain":"unknown"

{

"dev":"region2",

"size":"2478.25 GiB (2661.00 GB)",

"available_size":"2478.25 GiB (2661.00 GB)",

"max_available_extent":"2478.25 GiB (2661.00 GB)",

"type":"pmem",

"iset_id":"0x907ca37261be7a9b",

"persistence_domain":"unknown"

}

]

Step 4: Create namespaces each of which has the size same as the region size.

This can be done by omitting the --size argument in the create-namespace command.This creates a namespace spanning the entire free space available in the region.

For example, the following command creates a namespace on region4 of size 1636.39 GiB.

linux-6ivl:~ # ndctl create-namespace -r region4

{

"dev":"namespace4.0",

"mode":"fsdax",

"map":"dev",

"size":"1636.39 GiB (1757.06 GB)",

"uuid":"afba2267-b2cb-488a-8abd-8a46e07fe5dc",

"sector_size":512,

"align":16777216,

"blockdev":"pmem4"

}

Use the following command to create namspaces of size equal to the region size on all eight regions.

#!/bin/sh

for reg in `seq 0 7`

ndctl create-namespace -r region$reg

done

linux-6ivl:~ #

{

"dev":"namespace0.0",

"mode":"fsdax",

"map":"dev",

"size":"1279.73 GiB (1374.10 GB)",

"uuid":"e7999144-a556-40f5-8eff-41527fc1374c",

"sector_size":512,

"align":16777216,

"blockdev":"pmem0"

}

{

"dev":"namespace1.0",

"mode":"fsdax",

"map":"dev",

"size":"1703.58 GiB (1829.20 GB)",

"uuid":"4bbc90d2-e6ae-4c77-b893-1eaefcb10315",

"sector_size":512,

"align":16777216,

"blockdev":"pmem1"

}

{

"dev":"namespace2.0",

"mode":"fsdax",

"map":"dev",

"size":"2475.83 GiB (2658.40 GB)",

"uuid":"b964a277-ae56-40e3-a3d6-6942f673caa4",

"sector_size":512,

"align":16777216,

"blockdev":"pmem2"

}

{

"dev":"namespace3.0",

"mode":"fsdax",

"map":"dev",

"size":"2723.83 GiB (2924.69 GB)",

"uuid":"1ac0839b-8ca8-4bd4-82bc-71193bac658c",

"sector_size":512,

"align":16777216,

"blockdev":"pmem3"

}

{

"dev":"namespace4.0",

"mode":"fsdax",

"map":"dev",

"size":"1636.39 GiB (1757.06 GB)",

"uuid":"afba2267-b2cb-488a-8abd-8a46e07fe5dc",

"sector_size":512,

"align":16777216,

"blockdev":"pmem4"

}

{

"dev":"namespace5.0",

"mode":"fsdax",

"map":"dev",

"size":"2455.09 GiB (2636.14 GB)",

"uuid":"507fb4cf-caf9-4aa8-babc-06cd9493bc4c",

"sector_size":512,

"align":16777216,

"blockdev":"pmem5"

}

{

"dev":"namespace6.0",

"mode":"fsdax",

"map":"dev",

"size":"2455.34 GiB (2636.41 GB)",

"uuid":"e2cb67f2-db38-46ef-ab17-764d0c19cddd",

"sector_size":512,

"align":16777216,

"blockdev":"pmem6"

}

{

"dev":"namespace7.0",

"mode":"fsdax",

"map":"dev",

"size":"1638.14 GiB (1758.94 GB)",

"uuid":"ef56f07e-fa76-4743-8341-153d41a48871",

"sector_size":512,

"align":16777216,

"blockdev":"pmem7"

}

This is an example namespace created to demonstrate device mapper creation on persistent device memory. Users can always create multiple namespaces within a single region.

Step 5: Create physical volume on all the persistent memory devices.

Run the following script to create physical volumes (0 to 7) on persistent memory devices.

#!/bin/sh

# pvcreate on pmem devices

for pv in `seq 0 7`

pvcreate /dev/pmem$pv

done

Note that the physical volumes are created on the physical memory devices (0 to 7).

linux-6ivl:~ #

Physical volume "/dev/pmem0" successfully created.

Physical volume "/dev/pmem1" successfully created.

Physical volume "/dev/pmem2" successfully created.

Physical volume "/dev/pmem3" successfully created.

Physical volume "/dev/pmem4" successfully created.

Physical volume "/dev/pmem5" successfully created.

Physical volume "/dev/pmem6" successfully created.

Physical volume "/dev/pmem7" successfully created.

List the newly created physical volumes.

linux-6ivl:~ # pvs

PV VG Fmt Attr PSize PFree

/dev/mapper/3600507680281059a0800000000000122-part2 system lvm2 a-- 39.99g 0

/dev/mapper/3600507680281059a080000000000016b-part2 lvm2 --- 39.99g 39.99g

/dev/pmem0 lvm2 --- 1.25t 1.25t

/dev/pmem1 lvm2 --- 1.66t 1.66t

/dev/pmem2 lvm2 --- 2.42t 2.42t

/dev/pmem3 lvm2 --- 2.66t 2.66t

/dev/pmem4 lvm2 --- 1.60t 1.60t

/dev/pmem5 lvm2 --- 2.40t 2.40t

/dev/pmem6 lvm2 --- 2.40t 2.40t

/dev/pmem7 lvm2 --- 1.60t 1.60t

Step 6: Add the physical volumes to a volume group.

Run the following command to create a new volume group, named PmemVol, and add the physical volumes (0 to 7) to the volume group.

linux-6ivl:~ # vgcreate PmemVol /dev/pmem0 /dev/pmem1 /dev/pmem2 /dev/pmem3 /dev/pmem4 /dev/pmem5 /dev/pmem6 /dev/pmem7

Volume group "PmemVol" successfully created

Run the pvs command to verify if the physical volumes are now associated with the new volume group, PmemVol.

linux-6ivl:~ # pvs

PV VG Fmt Attr PSize PFree

/dev/mapper/3600507680281059a0800000000000122-part2 system lvm2 a-- 39.99g 0

/dev/mapper/3600507680281059a080000000000016b-part2 lvm2 --- 39.99g 39.99g

/dev/pmem0 PmemVol lvm2 a-- 1.25t 1.25t

/dev/pmem1 PmemVol lvm2 a-- 1.66t 1.66t

/dev/pmem2 PmemVol lvm2 a-- 2.42t 2.42t

/dev/pmem3 PmemVol lvm2 a-- 2.66t 2.66t

/dev/pmem4 PmemVol lvm2 a-- 1.60t 1.60t

/dev/pmem5 PmemVol lvm2 a-- 2.40t 2.40t

/dev/pmem6 PmemVol lvm2 a-- 2.40t 2.40t

/dev/pmem7 PmemVol lvm2 a-- 1.60t 1.60t

List the persistent memory volume group.

linux-6ivl:~ # vgs

VG #PV #LV #SN Attr VSize VFree

PmemVol 8 0 0 wz--n- 15.98t 15.98t

system 1 2 0 wz--n- 39.99g 0

linux-6ivl:~ # vgdisplay PmemVol

--- Volume group ---

VG Name PmemVol

System ID

Format lvm2

Metadata Areas 8

Metadata Sequence No 1

VG Access read/write

VG Status resizable

MAX LV 0

Cur LV 0

Open LV 0

Max PV 0

Cur PV 8

Act PV 8

VG Size 15.98 TiB

PE Size 4.00 MiB

Total PE 4190184

Alloc PE / Size 0 / 0

Free PE / Size 4190184 / 15.98 TiB

VG UUID K0DMQh-0SFa-k2ge-NWVr-zx9b-42J3-COTEub

Step 7: Create device mapper’s linear logical volumes.

To create device mapper’s linear volumes, you can run the lvcreate command with the new volume name along the the -n option, specify the size using the -L option, and specify the volume group on which to create the logical volume.

linux-6ivl:~ # lvcreate -n PmemDB -L 16000G PmemVol

Logical volume "PmemDB" created.

Step 8: Validate the logical volumes on the physical memory.

Run the following command to validate the newly created logical volumes, Universally Unique Identifier (UUID) of the pmem namespace, ndctl lists with UUID.

linux-6ivl:~ # ls -l /dev/mapper/PmemVol-*

lrwxrwxrwx 1 root root 7 Dec 11 09:50 /dev/mapper/PmemVol-PmemDB -> ../dm-8

linux-6ivl:~ # ls -l /dev/disk/by-id//pmem*

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-1ac0839b-8ca8-4bd4-82bc-71193bac658c -> ../../pmem3

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-4bbc90d2-e6ae-4c77-b893-1eaefcb10315 -> ../../pmem1

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-507fb4cf-caf9-4aa8-babc-06cd9493bc4c -> ../../pmem5

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-afba2267-b2cb-488a-8abd-8a46e07fe5dc -> ../../pmem4

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-b964a277-ae56-40e3-a3d6-6942f673caa4 -> ../../pmem2

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-e2cb67f2-db38-46ef-ab17-764d0c19cddd -> ../../pmem6

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-e7999144-a556-40f5-8eff-41527fc1374c -> ../../pmem0

lrwxrwxrwx 1 root root 11 Dec 11 09:50 /dev/disk/by-id//pmem-ef56f07e-fa76-4743-8341-153d41a48871 -> ../../pmem7

linux-6ivl:~ # ndctl list -N | jq ' .. | (.blockdev, .dev, .uuid)?'

"pmem4"

"namespace4.0"

"afba2267-b2cb-488a-8abd-8a46e07fe5dc"

"pmem6"

"namespace6.0"

"e2cb67f2-db38-46ef-ab17-764d0c19cddd"

"pmem1"

"namespace1.0"

"4bbc90d2-e6ae-4c77-b893-1eaefcb10315"

"pmem3"

"namespace3.0"

"1ac0839b-8ca8-4bd4-82bc-71193bac658c"

"pmem5"

"namespace5.0"

"507fb4cf-caf9-4aa8-babc-06cd9493bc4c"

"pmem7"

"namespace7.0"

"ef56f07e-fa76-4743-8341-153d41a48871"

"pmem0"

"namespace0.0"

"e7999144-a556-40f5-8eff-41527fc1374c"

"pmem2"

"namespace2.0"

"b964a277-ae56-40e3-a3d6-6942f673caa4"

Run the lsblk command and verify that the persistent memory volume is approximately of 16 TB in size.

linux-6ivl:~ # lsblk | grep -i pmem

pmem0 259:0 0 1.3T 0 disk