AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.

 View Only

Installing AIX on an FC-NVMe disk

By Jim Allen posted Mon November 09, 2020 05:27 PM

  


Updated December 7, 2022 (for Power 10, AIX 7.3, AIX 7.2 TL 5 SP5 changes, AIX 7.3 TL 1/VIOS 3.1.4 FC-NVMe NPIV changes)

AIX 7.2 TL 5 and AIX 7.3 on a Power 9 host (with FW 950 or later) or Power 10 host (FW 1010 or later) support installing the base operating system to FC-NVMe disks which are switch attached. This support is only provided for 32Gb FC adapters (EN1A, EN1B, EN1L, EN1M), and 64Gb FC adapters (EN1N, EN1P) which have at least the 12.4.257.30 (FCode level 12.8.9) adapter firmware level. FC-NVMe disk boot is also supported via NPIV from an AIX 7.3 TL1 LPAR with VIOS 3.1.4 for Power 10 (FW 1030 or later).

To enable access to FC-NVMe namespaces, the storage subsystem administrator must grant host access to the FC-NVMe disk by specifying a list of host NQNs that will be allowed. If the host NQNs are not provided to the storage subsystem and it is not allowing anyone in general to access the storage, then AIX will not be able to discover the disks and display them as valid candidates for installation in the Base Operation System (BOS) Install menus.

AIX 7.2 TL 5 (SP 1 thru SP4) uses a host NQN per FC (Fibre Channel) port that is derived from the FC port's World Wide Port Name (WWPN) and World Wide Node Name (WWNN). Similarly, Power 9 Partition Firmware also uses a host NQN per FC port based on its WWPN and WWNN. AIX 7.3, AIX 7.2 TL 5 SP5 (and later) and VIOS 3.1.4 use a host NQN per LPAR based on the partition UUID per LPAR.  Similarly, Power 10 Partition Firmware also uses this same host NQN (using partition UUID).  Thus for AIX 7.3, AIX 7.2 TL 5 SP5 (and later), VIOS 3.1.4,  and/or Power 10 Partition Firmware,  the same Host NQN is used for all FC ports assigned to that LPAR.  The AIX 7.3, AIX 7.2 TL 5 SP5 (and later) and VIOS 3.1.4 host NQN is identical to the Power 10 Partition Firmware host NQN, which allows just one host NQN for installing AIX on Power 10.  To enable boot on an FC-NVMe disk, both the AIX host NQNs and Partition Firmware host NQN(s) must be added to the storage subsystem's access list for each FC port for which boot will be supported. If an FC-NVMe disk is used by AIX, but not as member of the rootvg, then there is no need for that disk to include the Partition FW host NQN(s). The AIX host NQN is sufficient. 

AIX

On AIX, each FC port is displayable via the lsdev command with a name that starts with the string “fcs”. So for example fcs0 is the device name associated with an FC port. AIX will create children devices of the fcs device for each protocol they support: fscsi (FC SCSI protocol device), and fcnvme (FC-NVMe protocol device). The fcnvme device's host_nqn attribute contains the host NQN for that corresponding FC adapter. The fcnvme devices only exist as children on FC adapters which support FC-NVMe. To display all attributes of an fcnvme device, use the lsattr command. Thus to display all attributes of the fcnvme0 device, one would use the command: lsattr -El fcnvme0. Here is an example:

lsattr -El fcnvme0
attach switch How this adapter is connected False
autoconfig available Configuration State True
host_nqn nqn.2018-04.com.ibm:aix.fc.pn.0x100000109b2189fa.nn.0x200000109b2189fa Host NQN (NVMe Qualified Name) True

Thus the host NQN in this example
is nqn.2018-04.com.ibm:aix.fc.pn.0x100000109b2189fa.nn.0x200000109b2189fa.

If one is instead in the AIX BOS Install menu, then one needs to temporarily select the maintenance shell to view this information. With the default path environment in the maintenance shell one can run the odmget command to retrieve this information

(odmget -q “name=fcnvme0 AND attribute=host_nqn” CuAt). For example:

odmget -q "name=fcnvme0 AND attribute=host_nqn" CuAt

CuAt:
    name = "fcnvme0"

    attribute = "host_nqn"
    value = "nqn.2018-04.com.ibm:aix.fc.pn.0x100000109b2189fa.nn.0x200000109b2189fa"
    type = "R"
    generic = "DU"
    rep = "s"
    nls_index = 6

 

Alternatively if one knows the WWPN and WWNN of the FC port (perhaps from a label on the FC adapter, or from information displayed from the switch menus), one can also derive the AIX 7.2 TL 5 (SP1- SP4) host NQN for the FC port. The AIX 7.2 TL 5 (SP1- SP4) host NQN has the following format:

    nqn.2018-04.com.ibm:aix.fc.pn.<WWPN>.nn.<WWNN>

    where <WWPN> is the World Wide Port Name of that port with the prefix 0x, and <WWNN> is the World Wide Node Name of that port
    with the prefix 0x

For AIX 7.3, AIX 7.2 TL5 SP5 (and later) and VIOS 3.1.4 each fcvnme device's host_nqn attribute will have the same host NQN, which has the following format:

nqn.2014-08.org.nvmexpress:uuid:<partition_UUID>
where <partition_UUID> is the partition UUID for that LPAR.

The partition UUID can be found from the following AIX command:  lsattr -El sys0 -a partition_uuid, which will show an output similar to the following:

partition_uuid 58afdbc5-ce61-4406-a508-826ff1fc3efd Partition UUID False


Updating from AIX 7.2 TL 5 (SP1 - SP4) to AIX 7.2 TL 5 SP5 (or later)

If one does a update install from AIX 7.2 TL 5 (SP 1 - SP 4) to AIX 7.2 TL 5 SP5 (or later), then the the host NQNs assigned in AIX 7.2 TL5 (SP1 - SP4) will be preserved. (i.e. one NQN per FC port) Any new FC HBAs added after the update, will use the new AIX 7.2 TL 5 SP 5 format (i.e. one host NQN per LPAR using the partition ID). 

Migration from AIX 7.2 TL 5 to AIX 7.3

If one does a migration install from AIX 7.2 TL 5 to AIX 7.3, then the migration install will update all fcnvme device host NQN attributes on that LPAR to the new AIX 7.3 format (i.e. one host NQN per LPAR using the partition ID).  Thus if one is using an FC-NVMe boot disk, the storage subsystem needs to add the new AIX 7.3 host NQN to prior to the migration install. After the migration install the old AIX host NQNs per FC port can be removed from the storage subsystem's host access list. 

Partition Firmware

Partition Firmware displays its host NQN per FC port via the SMS IO Device Information menus. On the main SMS menu, select the I/O Device Information option which will display the following menu then select the SAN option, then select the FC NVMe over Fabric option:

1) SAN
2) FC NVMe over Fabric

The “FC NVMe over Fabric” option will display the Partition FW host NQNs for each FC adapter port. For Power 10, the Partition FW host NQN will be the same for all FC adapter ports assigned to an LPAR, where in Power 9 they are unique per FC adapter port. The below example shows Power 9 host NQN format.

PowerPC Firmware
Version FW950.00 (VL950_021)
SMS (c) Copyright IBM Corp. 2000,2020 All rights reserved.

-------------------------------------------------------------------------------
Select Media Adapter
1. U78D2.001.WZS0012-P1-C3-T1  /pci@800000020000022/fibre-channel@0

Host NQN: nqn.2020-02.com.ibm:powervm.pfw.pn.0x100000109b170787.nn.0x200000109b170787

2. U78D2.001.WZS0012-P1-C3-T2  /pci@800000020000022/fibre-channel@0,1
 Host NQN: nqn.2020-02.com.ibm:powervm.pfw.pn.0x100000109b170788.nn.0x200000109b170788

Also when one uses the SMS menu to display information about a specific NVMe namespace (disk), it will also display the Partition FW host NQN: 

PowerPC Firmware
Version FW950.00 (UL950_013)
SMS (c) Copyright IBM Corp. 2000,2020 All rights reserved.

-------------------------------------------------------------------------------
Namespace Details

U78D2.001.RCH0012-P1-C7-L1         745 GB Namespace - bootable
Host NQN: nqn.2020 02.com.ibm:powervm.pfw.pn.0x100000109b170787.nn.0x200000109b170787

Alternatively for Power 9 if one knows the WWPN and WWNN of the FC port (as in the AIX case above), one can construct the Power 9 Partition FW host NQN without using the SMS menus. The Power 9 Partition FW host NQN has the following format:

nqn.2020-02.com.ibm:powervm.pfw.pn.<WWPN>.nn.<WWNN>

where <WWPN> is the World Wide Port Name of that port with the prefix 0x, and <WWNN> is the World Wide Node Name of that port with the prefix 0x

The AIX 7.3,  AIX 7.2 TL 5 SP5 (and later), VIOS 3.1.4,  and Power 10 Partition FW host NQN have the following format:

nqn.2014-08.org.nvmexpress:uuid:<partition_UUID>

where <partition_UUID> is the partition UUID for that LPAR. For an AIX LPAR, the partition UUID can be found from the following command:  lsattr -El sys0 -a partition_uuid, which will show an output similar to the following:

partition_uuid 58afdbc5-ce61-4406-a508-826ff1fc3efd Partition UUID False

 

Storage Firmware

IBM FC-NVMe storage subsystems such as the FS 9200 require the 8.4.0.0 or higher firmware level to support FC-NVMe boot.

6 comments
200 views

Permalink

https://community.ibm.com/community/user/blogs/jim-allen1/2020/11/09/installing-aix-on-an-fc-nvme-disk

Comments

Jim Allen
Wed January 18, 2023 10:35 PM

Hi Mark,

IBMi does not support FC-NVMe.

MARK HAIS
Sun December 11, 2022 06:41 AM

Does IBM i LPAR supports FC-NVMe through VIOS. If yes, what are the prereq - VIOS ver, IBM i ver, etc.. ?
Thanks.

Jim Allen
Thu December 08, 2022 11:22 AM

Hi Harald,

Now that VIOS 3.1.4 and AIX 7.3 TL 1 have been released, I updated the above blog to reflect support for FC-NVMe NPIV boot for AIX.

Harald Stürner
Fri November 04, 2022 07:55 AM

Hello, with VIOS 3.1.4 we got following announcement:

IBM VIOS 3.1.4
  • Non-volatile memory express (NVMe) over NPIV with virtual FC
It would be great if you could update this post belonging to VIOS 3.1.4

Regards, Harald

Jim Allen
Wed May 25, 2022 11:15 AM

Hi Stefan,  Thanks for your question.  I can not comment on specific future plans, but I can say we are seriously evaluating FC-NVMe from AIX NPIV clients.  If we do add this, it will be part of the releases notes for the VIOS at that time. Also this blog will be updated at that time.

Stefan Koller
Wed May 25, 2022 10:27 AM

Hello 
Thanks a lot for this blog.

You wrote:
This is not supported via NPIV from an AIX LPAR: instead the FC adapter must be assigned to the AIX LPAR.

Do you think this limitation will fall in the future or is this not possible?
I am thinking of big Power Systems with many LPARs you cannot add a dedicated adapter to all LPARs.