iSCSI is an acronym for Internet Small Computer Systems Interface. It provides block-level access to storage devices by carrying SCSI commands over a TCP/IP network. iSCSI is used to facilitate data transfers over internet using TCP, a reliable transport mechanism using either IPV6 or IPV4 protocols.
iSCSI support in PowerVM
There was no direct support for using iSCSI attached devices by VIOS in the PowerVM architecture prior to VIOS 3.1.0. The VIOS 3.1.0 added changes both to VSCSI driver and also to AIX iSCSI device driver to support iSCSI disks in VIOS. The iSCSI support in VIOS allows iSCSI disks to be exported to client LPARs as virtual disks (vSCSI disks). This support is available in VIOS 3.1.0 onwards and requires FW 860.20 or later. The FW level of 860.20 is supported on P8 systems. For a P9 system, the minimum FW level is 910.
Support for using iSCSI disks with vSCSI requires that the discovery policy is not set to “file”, which is the default discovery policy. Users will need to change it “odm” before being able to use iSCSI disks with vSCSI. For LPAR Mobility, the destination VIOSes will need to be configured similar to the source side with respect to iSCSI and have access to the same storage.
VIOS 3.1 also enables MPIO support for the iSCSI initiator. With MPIO support, users will be able to configure and create multiple paths to an iSCSI disk similar to what is available and supported for other protocols. Booting the VIOS from an iSCSI disk is currently not supported.
Limitations
- There is no VIOS boot support using iSCSI disk.
- There is no support for flat file based discovery policy.
- There is no support for iSCSI based LV backed devices. .
- There is no support for Shared Storage Pools using iSCSI disks as either Repo or Shared pool disks.
- iSCSI disks or iSCSI based LVs or VGs cannot be used as paging devices for Active Memory Sharing (AMS) or Remote Restart support.
- If the backing device is an iSCSI disk, the client_reserve and mirrored attribute for the virtual target devices will not be supported.
Best Practices
For optimal performance, the following hardware configuration is suggested.
- A separate private network to access the iSCSI storage.
- Use of high speed network adapters and switches (at least 10G recommended).
- Before making any updates for maintenance on the primary VIOS that could cause a network outage (Example microcode update of the network adapter), ensure that the SEA is failed over to the secondary VIOS to avoid seeing errors accessing the iSCSI disks.
Configuration commands
Existing AIX users are familiar with the following commands to manage iscsi devices: lsiscsi, mkiscsi, chiscsi and rmiscsi.
These commands have been made available in VIOS to the padmin user via RBAC.
Example:
VIOS1: rchvios1
1. To view the attributes of the iscsi0 protocol device, run the following command. Please note that the disc_policy is set to “odm”. Also, make note of the initiator_name assigned by the AIX. This information and the VIOS ip address(es) are needed to do the configuration on the storage side. Please note that the assigned initiator name may be changed by the user to meet local iSCSI name requirements.
$ lsattr -El iscsi0
disc_filename /etc/iscsi/targets Configuration file False
disc_policy odm Discovery Policy True
initiator_name iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033 iSCSI Initiator Name True
isns_srvnames auto iSNS Servers IP Addresses True
isns_srvports iSNS Servers Port Numbers True
isw_err_recov fast_fail iSCSI Network Error Recovery Policy True
max_targets 16 Maximum Targets Allowed True
max_xfer_size 0x80000 Maximum transfer size True
num_cmd_elems 200 Maximum number of commands to queue to driver True
2. It is recommended to have a separate private network to access the iSCSI storage. You could have two subnets setup (10.10.10.x and 10.20.10.x in the example given below) to provide two iSCSI paths. We could configure SEA to provide the network access.
$ lsdev -dev ent*
ent0 Available 00-00 PCIe2 2-port 10GbE SR Adapter (a21910071410d003)
ent1 Available 00-01 PCIe2 2-port 10GbE SR Adapter (a21910071410d003)
ent2 Available 01-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent3 Available 01-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent4 Available 01-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent5 Available 01-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent6 Available Virtual I/O Ethernet Adapter (l-lan)
ent7 Available Virtual I/O Ethernet Adapter (l-lan)
ent8 Available Virtual I/O Ethernet Adapter (l-lan)
ent9 Available Virtual I/O Ethernet Adapter (l-lan)
ent10 Available Virtual I/O Ethernet Adapter (l-lan)
ent11 Available Virtual I/O Ethernet Adapter (l-lan)
ent12 Available Shared Ethernet Adapter
ent13 Available Shared Ethernet Adapter
ent14 Available Shared Ethernet Adapter
$ netstat -state
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en12 1500 link#2 6c.ae.8b.69.26.88 6697520 0 18314 0 0
en12 1500 9.5.160 9.5.160.51 6697520 0 18314 0 0
en13 1500 link#3 0.0.c9.db.c4.8a 15426089638 0 5416579881 0 0
en13 1500 10.10.10 10.10.10.10 15426089638 0 5416579881 0 0
en14 1500 link#4 0.0.c9.db.c4.8b 13464013 0 4580203 0 0
en14 1500 10.20.10 10.20.10.10 13464013 0 4580203 0 0
lo0 16896 link#1 119454 0 119454 0 0
lo0 16896 127 127.0.0.1 119454 0 119454 0 0
lo0 16896 ::1%1 119454 0 119454 0 0
3. Use the iSCSI commands to configure the iscsi0 protocol device and also access to iSCSI storage. Run the rmdev command on the iscsi0 and then the cfgmgr commands to discover the disks.
Note: The storage will also need to be configured to recognize the initiator IQN (iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033) and its IP addresses (10.10.10.10 and 10.20.10.10).
# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.1
# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.20.10.1
# chdev -l iscsi0 -a disc_policy=odm -a initiator_name=iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033
# rmdev -Rl iscsi0;cfgmgr -vl iscsi0
4. To check the iSCSI storage access configuration, run the following commands.
$ lsiscsi
iscsi0 10.10.10.1 3260 iqn.2005-10.com.xivstorage:06031903
iscsi0 10.20.10.2 3260 iqn.2005-10.com.xivstorage:06031903
$ lsdev -type disk
name status description
hdisk0 Available SAS Disk Drive
hdisk1 Available SAS Disk Drive
hdisk2 Available SAS RAID 0 Disk Array
hdisk3 Available SAS RAID 10 Disk Array
hdisk4 Available MPIO 2810 XIV Disk
hdisk5 Available MPIO 2810 XIV Disk
hdisk6 Available MPIO 2810 XIV Disk
hdisk7 Available MPIO 2810 XIV Disk
hdisk8 Available MPIO 2810 XIV Disk
hdisk9 Available MPIO 2810 XIV Disk
$ lsmpio
name path_id status path_status parent connection
dac0 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x0
dac0 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x0
hdisk0 0 Enabled Clo sas0 50000393e8110ac0,0
hdisk1 0 Enabled Clo sas0 50000393e811c264,0
hdisk2 0 Enabled Sel,Opt sas0 2070a1b100,0
hdisk3 0 Enabled Clo sas0 12c070a1b100,0
hdisk4 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x1000000000000
hdisk4 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x1000000000000
hdisk5 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x3000000000000
hdisk5 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x3000000000000
hdisk6 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x4000000000000
hdisk6 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x4000000000000
hdisk7 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x5000000000000
hdisk7 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x5000000000000
hdisk8 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x6000000000000
hdisk8 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x6000000000000
hdisk9 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x7000000000000
hdisk9 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x7000000000000
5. Repeat the above steps on the other VIOS as shown below.
VIOS2: rchvios2
$ lsattr -El iscsi0
disc_filename /etc/iscsi/targets Configuration file False
disc_policy odm Discovery Policy True
initiator_name iqn.com.ibm.stglabs.rch.rchvios2.hostid.0905a034 iSCSI Initiator Name True
isns_srvnames auto iSNS Servers IP Addresses True
isns_srvports iSNS Servers Port Numbers True
isw_err_recov fast_fail iSCSI Network Error Recovery Policy True
max_targets 16 Maximum Targets Allowed True
max_xfer_size 0x80000 Maximum transfer size True
num_cmd_elems 200 Maximum number of commands to queue to driver True
$ lsdev -dev ent*
ent0 Available 02-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent1 Available 02-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent2 Available 02-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent3 Available 02-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent4 Available 00-00 PCIe2 2-Port 10GbE Base-T Adapter (e4148e1614109204)
ent5 Available 00-01 PCIe2 2-Port 10GbE Base-T Adapter (e4148e1614109204)
ent6 Available Virtual I/O Ethernet Adapter (l-lan)
ent7 Available Virtual I/O Ethernet Adapter (l-lan)
ent8 Available Virtual I/O Ethernet Adapter (l-lan)
ent9 Available Virtual I/O Ethernet Adapter (l-lan)
ent10 Available Virtual I/O Ethernet Adapter (l-lan)
ent11 Available Virtual I/O Ethernet Adapter (l-lan)
ent12 Available Shared Ethernet Adapter
ent13 Available Shared Ethernet Adapter
ent14 Available Shared Ethernet Adapter
$ netstat -state
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en12 1500 link#2 98.be.94.5.7a.88 6717568 0 18451 0 0
en12 1500 9.5.160 9.5.160.52 6717568 0 18451 0 0
en13 1500 link#3 98.be.94.6f.ac.c0 9129522356 0 3264734055 0 0
en13 1500 10.10.10 10.10.10.11 9129522357 0 3264734055 0 0
en14 1500 link#4 98.be.94.6f.ac.c2 9150014155 0 3275326352 0 0
en14 1500 10.20.10 10.20.10.11 9150014155 0 3275326352 0 0
lo0 16896 link#1 280463 0 280463 0 0
lo0 16896 127 127.0.0.1 280463 0 280463 0 0
lo0 16896 ::1%1
# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.1
# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.2
# chdev -l iscsi0 -a disc_policy=odm
# rmdev -Rl iscsi0;cfgmgr -vl iscsi0
$ lsiscsi
iscsi0 10.10.10.1 3260 iqn.2005-10.com.xivstorage:06031903
iscsi0 10.20.10.2 3260 iqn.2005-10.com.xivstorage:06031903
$ lsdev -type disk
name status description
hdisk0 Available SAS 4K Disk Drive
hdisk1 Available SAS 4K Disk Drive
hdisk2 Available MPIO 2810 XIV Disk
hdisk3 Available MPIO 2810 XIV Disk
hdisk4 Available MPIO 2810 XIV Disk
hdisk5 Available MPIO 2810 XIV Disk
hdisk6 Available MPIO 2810 XIV Disk
hdisk7 Available MPIO 2810 XIV Disk
hdisk8 Available MPIO 2810 XIV Disk
hdisk9 Available MPIO 2810 XIV Disk
$ lsmpio
name path_id status path_status parent connection
dac0 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x0
dac0 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x0
hdisk0 0 Enabled Clo sas0 300,0
hdisk1 0 Enabled Sel sas0 1200,0
hdisk2 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x2000000000000
hdisk2 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x2000000000000
hdisk3 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x3000000000000
hdisk3 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x3000000000000
hdisk4 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x4000000000000
hdisk4 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x4000000000000
hdisk5 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x5000000000000
hdisk5 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x5000000000000
hdisk6 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x6000000000000
hdisk6 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x6000000000000
hdisk7 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x7000000000000
hdisk7 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x7000000000000
hdisk8 0 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x8000000000000
hdisk8 1 Enabled Clo iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x8000000000000
hdisk9 0 Enabled Sel,Opt iscsi0 iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x9000000000000
hdisk9 1 Enabled Sel,Opt iscsi0 iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x9000000000000
Diagnostic info
The error log on the VIOS captures any errors encountered for both iSCSI and vSCSI drivers. The snap command can be used capture the traces for VSCSI, Virtual Target devices backed by iSCSI disks and for the iSCSI driver.
References
For additional information on the supported iSCSI commands please refer to the following documentation:
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds3/lsiscsi.htm
For additional info on AIX iSCSI SW initiator documentation, please refer to the following documentation:
https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.networkcomm/iscsi_intro.htm
Contacting the PowerVM Team
Have questions for the PowerVM team or want to learn more? Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions