Contributors – Harsha Kotapati, Pravallika Chandolu, Yadagiri Rajaboina , Hanumanthu H and Nidhyanandhan
Introduction
The IBM NVMe expansion drawer(NED24) can hold up to 24 small form factor multipath capable dual port NVMe U.2 drives. The NVMe expansion drawer contains two Enclosure Services Managers (ESMs), connecting to the NVMe U.2 drives. With the POWER system Firmware level 1040, only one ESM is used to communicate with a NVMe U.2 drive. So NVMe U.2 drives are configured as single-path devices as explained with NED24 documentation .
What’s New
The POWER system Firmware level 1060.10 has enabled multipath capability, allowing both the ESMs to communicate with the dual port NVMe U.2 drives, as depicted in following Figure.
The Multipathing configuration provides improved resiliency and performance. For more details refer to the multipath NED24 documentation
IBM AIX operating system is enhanced to support multipath I/O capability with NVMe U.2 drives in NED24 configuration with following releases:
AIX Version 7.3 with the 7300-02 Technology Level and Service Pack 7300-02-02-2420, or later
AIX Version 7.2 with the 7200-05 Technology Level and Service Pack 7200-05-08-2420, or later
AIX Version 7.3 with the 7300-01 Technology Level and Service Pack 7300-01-04-2420, or later
NVMe drive representation and management
When using system Firmware level FW1060.10 or later, each NVMe U.2 drive in the NED24 NVMe expansion drawer has a "-R1" or "-R2" suffix added to the end of its physical location code. Both the drive paths must be assigned to a single logical partition(LPAR) using the HMC's logical partition Physical I/O Adapters page or through the partition’s profile. The Partner Location Code column can be used to identify the other path to the NVMe U.2 drive in a multipath configuration as shown in following figure:
Each drive path is represented as an independent PCIe endpoint. AIX enumerates them as independent adapter instances like nvme0 & nvme1 as shown below example:
For load balancing of I/O operations between the two drive paths, the namespaces should be created as shared and attached from both drive paths. The following example shows I/O paths of a shared namespace(hdisk0) from both drive ports accessible from ESM1 and ESM2.
For the creation of NVMe shared namespace in AIX, refer to the shared namespaces blog .
Self-Encrypting Drives(SED):
AIX supports encryption and decryption at drive level, hence SED state is applicable to the entire drive. Enabling or disabling SED on one of the drive paths will update the SED status of the other drive path also, as shown in below examples.
Verify the SED status of drive:
Enable SED on drive via one drive path, nvme0:
Verify SED status of drive, which shows SED Enabled status for both drive paths.
Verify disk label for encryption status
Hot-plug Operation:
The hot-plug procedure for PCIe adapter in AIX operating system requires the system administrator to take the PCIe adapter offline before starting the procedure. As each drive path is represented as independent PCIe endpoint, the hot plug remove operation on first adapter displays a message to run hot plug remove operation on the partner adapter as well.
The hot plug add operation of the partner controller does not instruct the user to place the physical drive in the slot as the drive is already placed during the hot plug add operation of device through first path.
References:
- https://www.ibm.com/docs/en/linux-on-systems?topic=servers-nvme-namespaces-ned24-nvme-expansion-drawer
- https://www.ibm.com/docs/en/power10?topic=POWER10/p10ham/p10ham_ned24_multipath.html
- https://community.ibm.com/community/user/power/blogs/yadagiri-rajaboina/2023/11/12/support-for-creating-nvme-shared-namespace-with-ai
End of the document