Originally posted by: Kimmel
This article summarises some support changes with regards to multipathing for two of the main distributed operating systems, AIX and MS Windows Server.
For many years, we've used additional driver add-ons for these to enable the full scope of multipathing with them, namely
- the Subsystem Device Driver Device Specific Module (SDDDSM) for MS Windows Server
- the Subsystem Device Driver Path Control Module (SDDPCM) for IBM AIX.
After the IBM Spectrum Virtualize products have declared in summer 2018 that they are going away from the above and go to the native multipath I/O functionality which is embedded in these respective operating systems:
https://www.ibm.com/support/pages/recommended-multi-path-driver-use-ibm-aix-and-vios-when-attached-svc-and-storwize-storage
also the DS8000 has declared likewise in summer 2019,
https://www.ibm.com/support/pages/sdddsm-and-sddpcm-end-support-ds8000
‒ and support for the extra SDDPCM and SDDDSM ceases end of this year.
POWER9 servers do not support SDDPCM anyway.
Multipathing functions like load balancing have now also moved over into the operating systems (instead of simple path failover), so going with native multipath I/O offers the advantage of quicker installation and not having the need to continuously update an extra driver add-on software.
The following is written out of a DS8000 perspective, and for AIX, but partially applies also to our other storage systems.
With SDDPCM in AIX, we had some exciting commands showing the physical path-to-port mapping at a DS8000, like this:
[p8-04v1:root:/home/root:] pcmpath query portmap
BAY-X BAY-Y
ESSID DISK H1 H2 H3 H4 H1 H2 H3 H4
ABCDEFGH ABCDEFGH ABCDEFGH ABCDEFGH ABCDEFGH ABCDEFGH ABCDEFGH ABCDEFGH
75ACA91 hdisk11 Bay3-Bay4: -------- -Y------ Y------- -------- --Y----- -------- -Y------ --------
75ACA91 hdisk12 Bay3-Bay4: -------- -Y------ Y------- -------- --Y----- -------- -Y------ --------
or like this:
[p8-04v1:root:/home/root:] pcmpath query essmap
Disk Path P Location adapter LUN SN Type Size LSS Vol Rank C/A S Connection port RaidMode
------- ----- - ------------ ------- ----------- ------------ ------- --- --- ---- --- - ----------- ---- --------
hdisk11 path0 00-00-02[FC] fscsi0 75ACA913200 IBM 2107-900 10.2GB 32 0 0009 12 Y R1-B4-H3-ZB 331 RAID6
hdisk11 path1 00-01-02[FC] fscsi1 75ACA913200 IBM 2107-900 10.2GB 32 0 0009 12 Y R1-B3-H2-ZB 211 RAID6
hdisk11 path2 01-00-02[FC] fscsi2 75ACA913200 IBM 2107-900 10.2GB 32 0 0009 12 Y R1-B3-H3-ZA 230 RAID6
hdisk11 path3 01-01-02[FC] fscsi3 75ACA913200 IBM 2107-900 10.2GB 32 0 0009 12 Y R1-B4-H1-ZC 302 RAID6
hdisk12 path0 00-00-02[FC] fscsi0 75ACA913201 IBM 2107-900 51.2GB 32 1 0009 12 Y R1-B4-H3-ZB 331 RAID6
hdisk12 path1 00-01-02[FC] fscsi1 75ACA913201 IBM 2107-900 51.2GB 32 1 0009 12 Y R1-B3-H2-ZB 211 RAID6
hdisk12 path2 01-00-02[FC] fscsi2 75ACA913201 IBM 2107-900 51.2GB 32 1 0009 12 Y R1-B3-H3-ZA 230 RAID6
hdisk12 path3 01-01-02[FC] fscsi3 75ACA913201 IBM 2107-900 51.2GB 32 1 0009 12 Y R1-B4-H1-ZC 302 RAID6
... and the above, and in that level of detail, are then no longer there.
Also a command like below, showing directly the port WWPN of each attached storage port, will look different in the future ‒
[p8-04v1:root:/home/root:] pcmpath query port
Active TargetPorts :6
Port# Wwpn State Mode Select Errors Paths Active
0 500507630318913e NORMAL ACTIVE 53119 0 20 20
1 500507630311513e NORMAL ACTIVE 15221 0 20 20
2 500507630313113e NORMAL ACTIVE 15135 0 20 20
3 50050763031b513e NORMAL ACTIVE 15166 0 20 20
4 500507630908d462 NORMAL ACTIVE 157188608 11 16 0
5 5005076309039462 NORMAL ACTIVE 157003611 13 16 0
A command like lsattr, applied to your hdisk, tells more about the multipathing software and algorithm used with it:
[p8-04v1:root:/home/root:] lsattr -El hdisk44
PCM PCM/friend/sddpcm PCM True
PR_key_value none Reserve Key True
algorithm load_balance Algorithm True
...
But we have additional command options now with the native AIX multipath I/O which at least allow a mapping to identify the storage system and volume serial number exactly that is currently used, and then commands in the DS8000 DSCLI like "lshostconnect -login" help to further identify at which ports a certain server is actually logged in.
Also the command "manage_disk_drivers -l" would tell you quickly whether you are on SDDPCM still ("NO_OVERRIDE"):
[p8-04v1:root:/home/root:] manage_disk_drivers -l
Device Present Driver Driver Options
2107DS8K NO_OVERRIDE NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
or already on native AIX MPIO ("AIX_AAPCM"):
[p9-04v2:root:/home/root:] manage_disk_drivers -l
Device Present Driver Driver Options
2107DS8K AIX_AAPCM NO_OVERRIDE,AIX_AAPCM,AIX_non_MPIO
And when on AIX multipath I/O, there are quite some options available to manage your volumes:
"lsmpio -q" showing the hdisks being DS8000 hdisks (=2107900), there sizes in GiB, and their DS8000 4-digit volume serial numbers.
[p9-04v2:root:/home/root:] lsmpio -q
Device Vendor Id Product Id Size Volume Name
---------------------------------------------------------------------------------
hdisk1 IBM 2107900 100.00GiB 6070
hdisk2 IBM 2107900 100.00GiB 6071
hdisk3 IBM 2107900 100.00GiB 6170
hdisk4 IBM 2107900 100.00GiB 6171
hdisk5 IBM 2107900 10.00GiB 6000
hdisk6 IBM 2107900 10.00GiB 6001
hdisk7 IBM 2107900 10.00GiB 6002
With the extended listing ("-q -l" on a hdisk), there is more information, and we can even identify the specific DS8000 machine this volume belongs to:
[p9-04v2:root:/home/root:] lsmpio -q -l hdisk2
Device: hdisk2
Vendor Id: IBM
Product Id: 2107900
Revision: 4.82
Capacity: 100.00GiB
Machine Type: 2107
Model Number: 980
Host Group: V11
Volume Name: 6071
Volume Serial: 6005076303FFD13E0000000000006071 (Page 83 NAA)
...since it indicates the WWPN 5005076303FFD13E of our DS8884 here, which in the DS8000, we get with the "showsi" command.
The "lsmpio -a" gives more information regarding the local fibre adapters used:
[p9-04v2:root:/home/root:] lsmpio -a
Adapter Driver: fscsi0 -> AIX PCM
Adapter WWPN: c05076068d1b004a
Link State: Up
Adapter Driver: fscsi1 -> AIX PCM
Adapter WWPN: c05076068d1b004c
Link State: Up
and it also comes with the full "-are" option, telling more about connection errors as well:
[p9-04v2:root:/home/root:] lsmpio -are
Adapter Driver: fscsi0 -> AIX PCM
Adapter WWPN: c05076068d1b004a
Link State: Up
Connection Errors
Last 10 Minutes: 0
Last 60 Minutes: 0
Last 24 Hours: 0
Total Errors: 0
Connection Errors
Last 10 Last 60 Last 24
Minutes Minutes Hours
50050763031b513e 0 0 0
Adapter Driver: fscsi1 -> AIX PCM
Adapter WWPN: c05076068d1b004c
Link State: Up
Connection Errors
Last 10 Minutes: 0
Last 60 Minutes: 0
Last 24 Hours: 0
Total Errors: 0
Connection Errors
Last 10 Last 60 Last 24
Minutes Minutes Hours
500507630311513e 0 0 0
The "-l" option when applied onto an hdisk shows exactly which paths are used, their status,
and what the port WWPNs of the storage system are for these.
[p9-04v2:root:/home/root:] lsmpio -l hdisk2
name path_id status path_status parent connection
hdisk2 0 Enabled Sel fscsi0 50050763031b513e,4060407100000000
hdisk2 1 Enabled fscsi1 500507630311513e,4060407100000000
A command like "lsmpio -S -l hdiskX -d" gives information about path errors, SCSI, and adapter errors.
The lsattr -El, applied to the hdisk, again tells more about the multipath tooling and algorithm used ‒ we see "fail_over" here.
[p9-04v2:root:/home/root:] lsattr -El hdisk2
DIF_prot_type none T10 protection type False
DIF_protection no T10 protection support True
FC3_REC false Use FC Class 3 Error Recovery True
PCM PCM/friend/aixmpiods8k Path Control Module False
PR_key_value none Persistant Reserve Key Value True+
algorithm fail_over Algorithm True+
...
queue_depth 20 Queue DEPTH True+
reserve_policy single_path Reserve Policy True+
...
unique_id 200B75ACA91607107210790003IBMfcp Unique device identifier False
ww_name 0x50050763031b513e FC World Wide Name False
Now the above outputs of lsmpio -S -l ... -d, lsmpio -l and lsattr -El all show me that of the two paths, it's just one single path used primarily, with the second one on standby only for failover. "fail_over" is the default algorithm with AIX-MPIO. When wanting to change this, then the chdev -a algorithm=shortest_queue command helps me, together with a short varyoffvg/varyonvg as needed:
[p9-04v2:root:/home/root:] varyoffvg ds8000vg
[p9-04v2:root:/home/root:] chdev -l hdisk16 -a algorithm=shortest_queue -a reserve_policy=no_reserve
hdisk16 changed
[p9-04v2:root:/home/root:] varyonvg ds8000vg
... and I see that both paths are selected now for active I/O, for instance with lsmpio, showing "Sel" in all rows:
[p9-04v2:root:/home/root:] lsmpio -l hdisk16
name path_id status path_status parent connection
hdisk16 0 Enabled Sel fscsi0 50050763031b513e,4060407200000000
hdisk16 1 Enabled Sel fscsi1 500507630311513e,4060407200000000
or also with lsattr -El,
[p9-04v2:root:/home/root:] lsattr -El hdisk16
DIF_prot_type none T10 protection type False
DIF_protection no T10 protection support True
FC3_REC false Use FC Class 3 Error Recovery True
PCM PCM/friend/aixmpiods8k Path Control Module False
PR_key_value none Persistant Reserve Key Value True+
algorithm shortest_queue Algorithm True+
...
reserve_policy no_reserve Reserve Policy True+
The link https://www.ibm.com/support/pages/how-migrate-sddpcm-aixpcm gives more guidance on how to migrate from SDDPCM to native AIX MPIO,
and https://www.ibm.com/support/knowledgecenter/ssw_aix_72/l_commands/lsmpio.html leads to the lsmpio Knowledge Center entry.