PowerVM

iSCSI support on PowerVM

By Veena Ganti posted Tue June 23, 2020 01:26 PM

  
Overview

iSCSI is an acronym for Internet Small Computer Systems Interface. It provides block-level access to storage devices by carrying SCSI commands over a TCP/IP network.  iSCSI is used to facilitate data transfers over internet using TCP, a reliable transport mechanism using either IPV6 or IPV4 protocols.

 

iSCSI support in PowerVM

There was no direct support for using iSCSI attached devices by VIOS in the PowerVM architecture prior to VIOS 3.1.0. The VIOS 3.1.0 added changes both to VSCSI driver and also to AIX iSCSI device driver to support iSCSI disks in VIOS. The iSCSI support in VIOS allows iSCSI disks to be exported to client LPARs as virtual disks (vSCSI disks).  This support is available in VIOS 3.1.0 onwards and requires FW 860.20 or later.  The FW level of 860.20 is supported on  P8 systems.  For a P9 system, the minimum FW level is 910.

Support for using iSCSI disks with vSCSI requires that the discovery policy is not set to “file”, which is the default discovery policy. Users will need to change it “odm” before being able to use iSCSI disks with vSCSI.  For LPAR Mobility, the destination VIOSes will need to be configured similar to the source side with respect to iSCSI and have access to the same storage.

VIOS 3.1 also enables MPIO support for the iSCSI initiator.  With MPIO support, users will be able to configure and create multiple paths to an iSCSI disk similar to what is available and supported for other protocols. Booting the VIOS from an iSCSI disk is currently not supported. 

Limitations

  • There is no VIOS boot support using iSCSI disk.
  • There is no support for flat file based discovery policy.
  • There is no support for iSCSI based LV backed devices.  .
  • There is no support for Shared Storage Pools using iSCSI disks as either Repo or Shared pool disks.
  • iSCSI disks or iSCSI based LVs or VGs  cannot be used as paging devices for Active Memory Sharing (AMS) or Remote Restart support.
  • If the backing device is an iSCSI disk, the client_reserve and mirrored attribute for the virtual target devices will not be supported.

Best Practices

For optimal performance, the following hardware configuration is suggested.

  • A separate private network to access the iSCSI storage.
  • Use of high speed network adapters and switches (at least 10G recommended).
  • Before making any updates for maintenance on the primary VIOS that could cause a network outage (Example microcode update of the network adapter), ensure that the SEA is failed over to the secondary VIOS to avoid seeing errors accessing the iSCSI disks.

Configuration commands

 

Existing AIX users are familiar with the following commands to manage iscsi devices: lsiscsi, mkiscsi, chiscsi and rmiscsi.

These commands have been made available in VIOS to the padmin user via RBAC.

 

Example:

 

VIOS1:    rchvios1

 

1. To view the attributes of the iscsi0 protocol device, run the following command. Please note that the disc_policy is set to “odm”.  Also, make note of the initiator_name assigned by the AIX.  This information and the VIOS ip address(es) are needed to do the configuration on the storage side. Please note that the assigned initiator name may be changed by the user to meet local iSCSI name requirements.

 

$ lsattr -El iscsi0

disc_filename  /etc/iscsi/targets                               Configuration file                            False

disc_policy    odm                                              Discovery Policy                              True

initiator_name iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033 iSCSI Initiator Name                          True

isns_srvnames  auto                                             iSNS Servers IP Addresses                     True

isns_srvports                                                   iSNS Servers Port Numbers                     True

isw_err_recov  fast_fail                                        iSCSI Network Error Recovery Policy           True

max_targets    16                                               Maximum Targets Allowed                       True

max_xfer_size  0x80000                                          Maximum transfer size                         True

num_cmd_elems  200                                              Maximum number of commands to queue to driver True

 

2. It is recommended to have a separate private network to access the iSCSI storage.  You could have two subnets setup (10.10.10.x and 10.20.10.x in the example given below) to provide two iSCSI paths. We could configure SEA to provide the network access.

 

$ lsdev -dev ent*

ent0  Available 00-00 PCIe2 2-port 10GbE SR Adapter (a21910071410d003)

ent1  Available 00-01 PCIe2 2-port 10GbE SR Adapter (a21910071410d003)

ent2  Available 01-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent3  Available 01-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent4  Available 01-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent5  Available 01-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent6  Available       Virtual I/O Ethernet Adapter (l-lan)

ent7  Available       Virtual I/O Ethernet Adapter (l-lan)

ent8  Available       Virtual I/O Ethernet Adapter (l-lan)

ent9  Available       Virtual I/O Ethernet Adapter (l-lan)

ent10 Available       Virtual I/O Ethernet Adapter (l-lan)

ent11 Available       Virtual I/O Ethernet Adapter (l-lan)

ent12 Available       Shared Ethernet Adapter

ent13 Available       Shared Ethernet Adapter

ent14 Available       Shared Ethernet Adapter

 

 

$ netstat -state

Name   Mtu   Network     Address                 Ipkts     Ierrs        Opkts     Oerrs  Coll

en12   1500  link#2      6c.ae.8b.69.26.88          6697520    0            18314     0     0

en12   1500  9.5.160     9.5.160.51                6697520     0            18314     0     0

en13   1500  link#3      0.0.c9.db.c4.8a       15426089638     0       5416579881     0     0

en13   1500  10.10.10    10.10.10.10           15426089638     0       5416579881     0     0

en14   1500  link#4      0.0.c9.db.c4.8b          13464013     0          4580203     0     0

en14   1500  10.20.10    10.20.10.10              13464013     0          4580203     0     0

lo0    16896 link#1                                 119454     0           119454     0     0

lo0    16896 127         127.0.0.1                  119454     0           119454     0     0

lo0    16896 ::1%1                                  119454     0           119454     0     0

 

3. Use the iSCSI commands to configure the iscsi0 protocol device and also access to iSCSI storage. Run the rmdev command on the iscsi0 and then the cfgmgr commands to discover the disks.

Note: The storage will also need to be configured to recognize the initiator IQN (iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033) and its IP addresses (10.10.10.10 and 10.20.10.10).

 

 

# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.1

# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.20.10.1

 

# chdev -l iscsi0 -a disc_policy=odm -a initiator_name=iqn.com.ibm.stglabs.rch.rchvios1.hostid.0905a033

# rmdev -Rl iscsi0;cfgmgr -vl iscsi0

 

4. To check the iSCSI storage access configuration, run the following commands.

 

$ lsiscsi

iscsi0       10.10.10.1    3260 iqn.2005-10.com.xivstorage:06031903    

iscsi0       10.20.10.2    3260 iqn.2005-10.com.xivstorage:06031903  

 

$ lsdev -type disk

name             status      description

hdisk0           Available   SAS Disk Drive

hdisk1           Available   SAS Disk Drive

hdisk2           Available   SAS RAID 0 Disk Array

hdisk3           Available   SAS RAID 10 Disk Array

hdisk4           Available   MPIO 2810 XIV Disk

hdisk5           Available   MPIO 2810 XIV Disk

hdisk6           Available   MPIO 2810 XIV Disk

hdisk7           Available   MPIO 2810 XIV Disk

hdisk8           Available   MPIO 2810 XIV Disk

hdisk9           Available   MPIO 2810 XIV Disk

 

$ lsmpio

name     path_id  status   path_status  parent  connection

 

dac0     0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x0

dac0     1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x0

hdisk0   0        Enabled  Clo          sas0    50000393e8110ac0,0

hdisk1   0        Enabled  Clo          sas0    50000393e811c264,0

hdisk2   0        Enabled  Sel,Opt      sas0    2070a1b100,0

hdisk3   0        Enabled  Clo          sas0    12c070a1b100,0

hdisk4   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x1000000000000

hdisk4   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x1000000000000

hdisk5   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x3000000000000

hdisk5   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x3000000000000

hdisk6   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x4000000000000

hdisk6   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x4000000000000

hdisk7   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x5000000000000

hdisk7   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x5000000000000

hdisk8   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x6000000000000

hdisk8   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x6000000000000

hdisk9   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x7000000000000

hdisk9   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x7000000000000

 

 

5. Repeat the above steps on the other VIOS as shown below.

 

VIOS2:    rchvios2

 

$ lsattr -El iscsi0

disc_filename  /etc/iscsi/targets                               Configuration file                            False

disc_policy    odm                                              Discovery Policy                              True

initiator_name iqn.com.ibm.stglabs.rch.rchvios2.hostid.0905a034 iSCSI Initiator Name                          True

isns_srvnames  auto                                             iSNS Servers IP Addresses                     True

isns_srvports                                                   iSNS Servers Port Numbers                     True

isw_err_recov  fast_fail                                        iSCSI Network Error Recovery Policy           True

max_targets    16                                               Maximum Targets Allowed                       True

max_xfer_size  0x80000                                          Maximum transfer size                         True

num_cmd_elems  200                                              Maximum number of commands to queue to driver True

 

 

$ lsdev -dev ent*

ent0  Available 02-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent1  Available 02-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent2  Available 02-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent3  Available 02-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)

ent4  Available 00-00 PCIe2 2-Port 10GbE Base-T Adapter (e4148e1614109204)

ent5  Available 00-01 PCIe2 2-Port 10GbE Base-T Adapter (e4148e1614109204)

ent6  Available       Virtual I/O Ethernet Adapter (l-lan)

ent7  Available       Virtual I/O Ethernet Adapter (l-lan)

ent8  Available       Virtual I/O Ethernet Adapter (l-lan)

ent9  Available       Virtual I/O Ethernet Adapter (l-lan)

ent10 Available       Virtual I/O Ethernet Adapter (l-lan)

ent11 Available       Virtual I/O Ethernet Adapter (l-lan)

ent12 Available       Shared Ethernet Adapter

ent13 Available       Shared Ethernet Adapter

ent14 Available       Shared Ethernet Adapter

 

$ netstat -state

Name   Mtu   Network     Address                 Ipkts     Ierrs        Opkts     Oerrs  Coll

en12   1500  link#2      98.be.94.5.7a.88          6717568     0            18451     0     0

en12   1500  9.5.160     9.5.160.52                6717568     0            18451     0     0

en13   1500  link#3      98.be.94.6f.ac.c0       9129522356     0       3264734055     0     0

en13   1500  10.10.10    10.10.10.11            9129522357     0       3264734055     0     0

en14   1500  link#4      98.be.94.6f.ac.c2       9150014155     0       3275326352     0     0

en14   1500  10.20.10    10.20.10.11            9150014155     0       3275326352     0     0

lo0    16896 link#1                                 280463     0           280463     0     0

lo0    16896 127         127.0.0.1                  280463     0           280463     0     0

lo0    16896 ::1%1                     

 

# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.1

# mkiscsi -l iscsi0 -g static -t iqn.2005-10.com.xivstorage:06031903 -n 3260 -i 10.10.10.2

 

# chdev -l iscsi0 -a disc_policy=odm

# rmdev -Rl iscsi0;cfgmgr -vl iscsi0

 

$ lsiscsi

iscsi0       10.10.10.1    3260 iqn.2005-10.com.xivstorage:06031903    

iscsi0       10.20.10.2    3260 iqn.2005-10.com.xivstorage:06031903  

$ lsdev -type disk

name             status      description

hdisk0           Available   SAS 4K Disk Drive

hdisk1           Available   SAS 4K Disk Drive

hdisk2           Available   MPIO 2810 XIV Disk

hdisk3           Available   MPIO 2810 XIV Disk

hdisk4           Available   MPIO 2810 XIV Disk

hdisk5           Available   MPIO 2810 XIV Disk

hdisk6           Available   MPIO 2810 XIV Disk

hdisk7           Available   MPIO 2810 XIV Disk

hdisk8           Available   MPIO 2810 XIV Disk

hdisk9           Available   MPIO 2810 XIV Disk

 

$ lsmpio

name     path_id  status   path_status  parent  connection

 

dac0     0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x0

dac0     1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x0

hdisk0   0        Enabled  Clo          sas0    300,0

hdisk1   0        Enabled  Sel          sas0    1200,0

hdisk2   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x2000000000000

hdisk2   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x2000000000000

hdisk3   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x3000000000000

hdisk3   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x3000000000000

hdisk4   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x4000000000000

hdisk4   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x4000000000000

hdisk5   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x5000000000000

hdisk5   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x5000000000000

hdisk6   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x6000000000000

hdisk6   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x6000000000000

hdisk7   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x7000000000000

hdisk7   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x7000000000000

hdisk8   0        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x8000000000000

hdisk8   1        Enabled  Clo          iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x8000000000000

hdisk9   0        Enabled  Sel,Opt      iscsi0  iqn.2005-10.com.xivstorage:06031903,10.10.10.1,0xcbc,0x9000000000000

hdisk9   1        Enabled  Sel,Opt      iscsi0  iqn.2005-10.com.xivstorage:06031903,10.20.10.2,0xcbc,0x9000000000000

 

Diagnostic info

 

The error log on the VIOS captures any errors encountered  for both iSCSI and vSCSI drivers. The snap command can be used capture the traces for VSCSI, Virtual Target devices backed by iSCSI disks and for the iSCSI driver.

References

For additional information on the supported iSCSI commands please refer to the following documentation:

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds3/lsiscsi.htm

 

For additional info on AIX iSCSI SW initiator documentation, please refer to the following documentation:

https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.networkcomm/iscsi_intro.htm

 


Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more? Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions
0 comments
61 views

Permalink