AIX

 View Only

AIX Virtual Fibre Channel Enhancement

By Chris Gibson posted Wed January 24, 2024 07:37 PM

  

 

AIX Virtual Fibre Channel Enhancement

In this post I want to discuss two topics. First, I want to tell you all about a new AIX 7.3 command called vfcstat. Second, I want to show some of the (not so recent) enhancements to the fcstat command.

New vfcstat command

A new Virtual Fibre Channel (VFC) client adapter enhancement was introduced with the release of AIX 7.3 Technology Level 2 Service Pack 1 (November 2023). A new command, called vfcstat, was introduced. This allows administrators to access information for VFC adapters, such as statistics, status, and activities.

Prior to this command, administrators could use the cat command to view the information stored in the /proc file system (in /proc/sys/adapter/fc/fcsX). The vfcstat command makes it easier for admins to view this information (without needing to know the location of the various files in /proc).

This command is available for AIX 7.3 systems that are Virtual I/O (VIO) clients, of VIOS running version 3.1.4.x and later (VIOS 3.1.4 provides support for NPIV multi-queue, https://www.ibm.com/support/knowledgecenter/POWER9/p9hb1/p9hb1_npiv_multi_queue_support.htm).

From the AIX 7.3 command reference for vfcstat:

“The vfcstat command reports information such as virtual FC adapter state, status, NPort ID, I/O queues information, read/write statistics, binding information with the host (VIOS), link errors, path timeouts, and so on. The required information is fetched in real time and displayed in the output. You can reset some of the details such as statistics and error counts. You can also retrieve read/write statistics in formatted output for scripting purposes by using the -F option. The vfcstat command displays the statistics in a specified time interval for the specific number of times. The vfcstat command works on virtual FC client adapter only.”

The new command is delivered with the devices.vdevice.IBM.vfc-client.rte fileset.

# lslpp -w /usr/bin/vfcstat

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/bin/vfcstat

                          devices.vdevice.IBM.vfc-client.rte        Symlink

# ls -ltr /usr/bin/vfcstat

lrwxrwxrwx    1 bin      bin              17 Oct 23 18:23 /usr/bin/vfcstat -> /usr/sbin/vfcstat

# lslpp -w /usr/sbin/vfcstat

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/sbin/vfcstat

                          devices.vdevice.IBM.vfc-client.rte        File

# lslpp -lc devices.vdevice.IBM.vfc-client.rte

#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked

/usr/lib/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:

/etc/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:

I tested the vfcstat command on an AIX system running 7300-02-01-2346.

You can use the command to list all the VFC adapters on your AIX system.

# vfcstat -l

fcs0

fcs1

To list all the functions available for the given VFC adapter, run the following:

# vfcstat -d fcs0

activity

capabilities

hostinfo

link

nport_id

status

tunables

wwpn

To display error counts for a given VFC adapter.

# vfcstat -d fcs0 -e

Current time                    : Jan 22 2024, 22:00:52.294

Time since last reset           : 5885149 sec ( 2 months 8 days 2 hours 45 minutes 49 seconds )

Link Errors:

           Total errors: 0

        Last 10 minutes: 0

        Last 60 minutes: 0

          Last 24 hours: 0

Path Timeouts:

                 Domain: Admin

         Total timeouts: 0

        Last 10 minutes: 0

        Last 60 minutes: 0

          Last 24 hours: 0

Path Timeouts:

                 Domain: SCSI

         Total timeouts: 0

        Last 10 minutes: 0

        Last 60 minutes: 0

          Last 24 hours: 0

Path Timeouts:

                 Domain: NVMeOF

         Total timeouts: 0

        Last 10 minutes: 0

        Last 60 minutes: 0

          Last 24 hours: 0

Note there are three domains for VFC adapter elements. One is for VFC Admin commands, the second is for SCSI commands and the third is for NVME commands.


Use the -s flag to display various statistics for a given VFC adapter. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/stats. We’ve included the –F option as it displays formatted statistic output (which is easier to read!).

# vfcstat -d fcs0 -s -F

Current time=Jan 22 2024, 22:38:58.102

Time since last reset=5887435 sec ( 2 months 8 days 3 hours 23 minutes 55 seconds )

Queue_Start

Domain=SCSI

Queue number=0

Read reqs=9814

Read bytes=196898541

Write reqs=2696118

Write bytes=253070213192

No DMA res count=0

No SGE count=0

h_send_dropped=0

h_send_closed=0

h_send_busy=0

No cmd res count=0

Queue_End

To display tunables of a given VFC adapter, use the -d and -f tunables options. This is same as reading proc file /proc/sys/adapter/fc/fcs0/tunables.

# vfcstat -d fcs0 -f tunables

lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)

max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)

num_cmd_elems   ( Requested / Granted ):     2048 / 2048

num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024

num_io_queues   ( Requested / Granted ):     8 / 0

num_nvme_queues ( Requested / Granted ):     4 / 0

To display VIOS host information of a given VFC adapter, use the -f hostinfo option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/hostinfo.

# vfcstat -d fcs0 -f hostinfo

VFC client adapter name                   : fcs0

Host partition name  (VIOS)               : vios1

VFC host adapter name                     : vfchost0

VFC host adapter location code            : U8286.42A.2153B1V-V1-C30

FC adapter name on VIOS                   : fcs0

FC adapter location code  on VIOS         : U78C9.001.WZS00XE-P1-C1-T1

To display WWPN information of a given VFC adapter, use the -f wwpn option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/wwpn.

# vfcstat -d fcs0 -f wwpn

0xc852100000000011

To reset all fields in the stats function for a given VFC adapter.

# vfcstat -d fcs0 -s –z

To reset all error counts for a given VFC adapter.

# vfcstat -d fcs0 -e –z

Here's a little script I wrote to collect and display output from each of the available functions for all VFC adapters on a system:

myvfcs=$(/usr/bin/vfcstat -l)

vfcsfuncs=$(echo $myvfcs | awk '{print $1}' | xargs /usr/bin/vfcstat -d)

echo Displaying functions that are available for a virtual FC client adapter: $myvfcs

echo

echo Functions available for $myvfcs are:

echo $vfcsfuncs

echo

echo

for xvfcs in $myvfcs

do

  for eachvfunc in $vfcsfuncs

  do

    echo "VFCSTAT $eachvfunc for $xvfcs" | xargs /usr/local/bin/box

    /usr/bin/vfcstat -d $xvfcs -f $eachvfunc

    echo

  done

done

The output is:

# ./xvfcs

Displaying functions that are available for a virtual FC client adapter: fcs0 fcs1

Functions available for fcs0 fcs1 are:

activity capabilities hostinfo link nport_id status tunables wwpn

#############################

# VFCSTAT activity for fcs0 #

#############################

Admin Queue

Admin I/O Queue

#################################

# VFCSTAT capabilities for fcs0 #

#################################

Requested capabilities         : CAN_MIGRATE, CAN_HANDLE_FPIN

Host responded capabilities    :

#############################

# VFCSTAT hostinfo for fcs0 #

#############################

VFC client adapter name                   : fcs0

Host partition name  (VIOS)               : vios1

VFC host adapter name                     : vfchost0

VFC host adapter location code            : U8286.42A.2153B1V-V1-C30

FC adapter name on VIOS                   : fcs0

FC adapter location code  on VIOS         : U78C9.001.WZS00YD-P1-C7-T1

#########################

# VFCSTAT link for fcs0 #

#########################

Link       UP

topology   FABRIC

speed      8Gb

#############################

# VFCSTAT nport_id for fcs0 #

#############################

0xe016a0

###########################

# VFCSTAT status for fcs0 #

###########################

State                     0x10 (ONLINE)

Opened                    true

cfg_opn_state             0x0

prev_cfg_channels         0

flags                     0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )

total_io_dma_size         0x20000000

total_dma_size_used       0x9900000

no_chs_reason_code        0x2 ( NO_PROCESSOR_COMPATABILITY )

migrated_count            0

online_time               373 (lbolt)

UP time                   5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )

Admin Queue

    active_cmds           0

    spl_active_cmds       0

    pending_cmds          0

    spl_pending_cmds      0

    resp_cmds             0

Admin I/O Queue

    active_cmds           0

    spl_active_cmds       0

    pending_cmds          0

    spl_pending_cmds      0

    waiting_cmds          0

    resp_cmds             0

    offlevel_resp_cmds    0

    cancel_resp_cmds      0

#############################

# VFCSTAT tunables for fcs0 #

#############################

lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)

max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)

num_cmd_elems   ( Requested / Granted ):     2048 / 2048

num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024

num_io_queues   ( Requested / Granted ):     8 / 0

num_nvme_queues ( Requested / Granted ):     4 / 0

#########################

# VFCSTAT wwpn for fcs0 #

#########################

0xc852100000000011

#############################

# VFCSTAT activity for fcs1 #

#############################

Admin Queue

Admin I/O Queue

#################################

# VFCSTAT capabilities for fcs1 #

#################################

Requested capabilities         : CAN_MIGRATE, CAN_HANDLE_FPIN

Host responded capabilities    :

#############################

# VFCSTAT hostinfo for fcs1 #

#############################

VFC client adapter name                   : fcs1

Host partition name  (VIOS)               : vios2

VFC host adapter name                     : vfchost0

VFC host adapter location code            : U8286.42A.2153B1V-V2-C30

FC adapter name on VIOS                   : fcs0

FC adapter location code  on VIOS         : U78C9.001.WZS00XE-P1-C1-T1

#########################

# VFCSTAT link for fcs1 #

#########################

Link       UP

topology   FABRIC

speed      8Gb

#############################

# VFCSTAT nport_id for fcs1 #

#############################

0xe016c0

###########################

# VFCSTAT status for fcs1 #

###########################

State                     0x10 (ONLINE)

Opened                    true

cfg_opn_state             0x0

prev_cfg_channels         0

flags                     0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )

total_io_dma_size         0x20000000

total_dma_size_used       0x9900000

no_chs_reason_code        0x2 ( NO_PROCESSOR_COMPATABILITY )

migrated_count            0

online_time               373 (lbolt)

UP time                   5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )

Admin Queue

    active_cmds           0

    spl_active_cmds       0

    pending_cmds          0

    spl_pending_cmds      0

    resp_cmds             0

Admin I/O Queue

    active_cmds           0

    spl_active_cmds       0

    pending_cmds          0

    spl_pending_cmds      0

    waiting_cmds          0

    resp_cmds             0

    offlevel_resp_cmds    0

    cancel_resp_cmds      0

#############################

# VFCSTAT tunables for fcs1 #

#############################

lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)

max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)

num_cmd_elems   ( Requested / Granted ):     2048 / 2048

num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024

num_io_queues   ( Requested / Granted ):     8 / 0

num_nvme_queues ( Requested / Granted ):     4 / 0

#########################

# VFCSTAT wwpn for fcs1 #

#########################

0xc852100000000013

Enhancements to fcstat

In a somewhat related set of enhancements, the fcstat command was also updated with several new options that provide additional information to the administrator. You may or may not be aware, but these new options have been available for some time with AIX 7.3. The new options are:

·      -p Displays a time-series report of traffic statistics for a specific transport protocol (TP) that is specified with the Protocol parameter.

·      -t Displays a time-series report of the traffic statistics continuously with a time interval between two consecutive reports, which is specified with the Interval parameter.

·      -e Displays all the statistics, that includes the device-specific statistics such as driver statistics, link statistics, FC4 types, and vital Small Form-factor Pluggable (SFP) parameters.

·      -d Displays the statistics by opening the adapter in diagnostic mode.

Here’s an example of using the -t and -p options. In the example below we have chosen to view the SCSI protocol statistics for the fcs0 adapter and to display the time-series report of traffic statistics with a one second interval.

# fcstat -t 1 -p scsi fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: Virtual Fibre Channel Client Adapter (adapter/vdevice/IBM,vfc-client)

TP         Read Reqs(K)     Write Reqs(K)   Read (GB)  Write (GB)

===== ================= ================= =========== ===========

SCSI                  9              2314           0         203

SCSI                  9              2315           0         203

SCSI                  9              2316           0         203

SCSI                  9              2317           0         203

SCSI                  9              2318           0         203

SCSI                  9              2319           0         203

SCSI                  9              2320           0         203

SCSI                  9              2322           0         204

SCSI                  9              2323           0         204

SCSI                  9              2324           0         204

SCSI                  9              2326           0         204

SCSI                  9              2328           0         204

SCSI                  9              2329           0         205

SCSI                  9              2330           0         205

SCSI                  9              2332           0         205

SCSI                  9              2334           0         205

SCSI                  9              2335           0         205

SCSI                  9              2336           0         206

SCSI                  9              2338           0         206

SCSI                  9              2340           0         206

SCSI                  9              2341           0         206

SCSI                  9              2342           0         206

SCSI                  9              2344           0         207

SCSI                  9              2346           0         207

SCSI                  9              2347           0         207

SCSI                  9              2348           0         207

SCSI                  9              2350           0         207

SCSI                  9              2352           0         208

SCSI                  9              2353           0         208

SCSI                  9              2354           0         208

In the output above the time-series report shows the following statistics:

·      Read Reqs(K): The number of read or input requests, in thousands.

·      Write Reqs(K): The number of write or output requests, in thousands.

·      Read(GB): The number of input bytes that are read, in gigabytes.

·      Write(GB): The number of output bytes that are written, in gigabytes.

If you do not have any NVME adapters/devices on your AIX system (or you are only interested in FC SCSI traffic), you can remove the -p flag and shorten the command to, fcstat -t 1 fcs0.

The -e option displays vital Small Form-factor Pluggable (SFP) parameters such as vendor information, temperature, voltage information, as well as, the device-specific statistics such as driver statistics, link statistics, and FC4 protocol types.

Previously, some of this information was only available using a special tool (called emfc_power) provided by IBM support. This option is useful with physical (not virtual) fibre channel adapters. It appears this enhancement may have been delivered based on this customer enhancement request, https://ibm-power-systems.ideas.ibm.com/ideas/AIX-I-688.

# fcstat -e fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: PCIe3 2-Port 32Gb FC Adapter (df1000e314101506) (adapter/pciex/df1000e31410150)

...etc...

FC SFP Information

  Vendor Name: AVAGO

  Vendor OUI:  0000176a

  Vendor PN:   AFBR-57G5MZ-ELX

  Temperature: 42.371 C     [Range -128 C - +128 C]

  Voltage:     3.331 V      [Range 0 V - +6.55 V]

  TX Bias:     5.466 mA     [Range 0 mA - 131 mA]

  TX Power:   -1.4837 dBm   [Range -40 dBm - +8.2 dBm]

               0.7106 mW    [Range 0 mW - 6.5535 mW]

  RX Power:   -3.6866 dBm   [Range -40 dBm - +8.2 dBm]

               0.4279 mW    [Range 0 mW - 6.5535 mW ]

                                      

The fcstat command opens the device in diagnostic mode by using the -d flag, as shown below. When the link is down and the device is opened in nondiagnostic mode, the fcstat command delays in generating the output. You can use the -c flag to remove this delay. If the device is already opened and the fcstat command is started with the -d flag, the open operation on the device fails with an EACCESS error. Meaning you can now run fcstat on an adapter, with a link that is down, and still obtain some useful information relating to link status, failure, and loss of sync/signal counts.

# fcstat -d fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03) (adapter/pciex/df1000f114108a0)

Serial Number: 1A202029F7

Option ROM Version: 0278213D

ZA: U2D2.10X13

World Wide Node Name: 0x20000000C9D55D1E

World Wide Port Name: 0x10000000C9D55D1E

FC-4 TYPES:

  Supported: 0x0000012000000000000000000000000000000000000000000000000000000000

  Active:    0x0000010000000000000000000000000000000000000000000000000000000000

Class of Service: 3

Port Speed (supported): 8 GBIT

Port Speed (running):   0 GBIT

Port FC ID: 0x000000

Port Type: UNKNOWN

Attention Type:   Link Down

Topology:  UNKNOWN

Seconds Since Last Reset: 0

        Transmit Statistics     Receive Statistics

        -------------------     ------------------

Frames: 0                       0

Words:  0                       0

LIP Count: 0

NOS Count: 0

Error Frames:  0

Dumped Frames: 0

Link Failure Count: 4

Loss of Sync Count: 4

Loss of Signal: 5

Primitive Seq Protocol Error Count: 0

Invalid Tx Word Count: 0

Invalid CRC Count: 0

FC SCSI Adapter Driver Information

  No DMA Resource Count: 0

  No Adapter Elements Count: 0

  No Command Resource Count: 0

FC SCSI Traffic Statistics

  Input Requests:   0

  Output Requests:  0

  Control Requests: 0

  Input Bytes:  0

  Output Bytes: 0

You can read more about these new options for fcstat in this blog from Yadagiri Rajaboina from IBM AIX Storage Drivers Development.

Displaying the Fibre channel HBA SFP information with AIX/VIOS

https://community.ibm.com/community/user/power/blogs/yadagiri-rajaboina/2023/10/29/displaying-fc-hba-sfp-information?CommunityKey=daa942cb-b783-4fd3-ba27-a2d7462f9530

I encourage you to read the command reference guides, and also the “What’s new” links below, to learn more about the latest enhancements.

References:

vfcstat Command Reference

https://www.ibm.com/docs/en/aix/7.3?topic=v-vfcstat-command

fcstat Command Reference

https://www.ibm.com/docs/en/aix/7.3?topic=f-fcstat-command

What's new in Commands - November 2023

https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__3

What's new in Commands - November 2022

https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__4


#power-featured-area-3

3 comments
74 views

Permalink

Comments

Mon February 12, 2024 08:27 AM

Hi Chris,

after FC driver upgrade to the latest "12.4.257.30" all works fine.

I had some old firmware 1.1.58.4 in place before.

Thanks!

Sun February 11, 2024 05:53 PM

Hi Pavel, thanks for your comment.

Can you please check that your 16Gb FC adapter has the latest version of firmware installed.

If the adapter is running an older firmware level, it is very likely that 'fcstat -e' will not present any useful information. The FC driver queries SFP parameters by issuing mailbox commands to adapter firmware. The old firmware level may not respond to mailbox requests and hence we see all zero’s for SFP parameters.

In our lab we updated our 16Gb FC adapter firmware to the latest available level i.e 12.4.257.30 and observed that ‘fcstat -e fcsX’ started reporting SFP stats correctly.

# lsdev | grep fcs0
fcs0       Available 01-00       PCIe3 2-Port 16Gb FC Adapter (df1000e21410f103)

# lsmcode -A | grep fcs0
fcs0!df1000e21410f103.00012000040025700030

# fcstat -e fcs0 | grep ZA
ZA: 12.4.257.30

# fcstat -e fcs0

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: PCIe3 2-Port 16Gb FC Adapter (df1000e21410f103) (adapter/pciex/df1000e21410f10)
Serial Number: 1A5480036E

ZA: 12.4.257.30
...
FC SFP Information
  Vendor Name: EMULEX
  Vendor OUI:  0000176a
  Vendor PN:   AFBR-57F5MZ-ELX
  Temperature: 43.629 C     [Range -128 C - +128 C]
  Voltage :    3.287 V      [Range 0 V - +6.55 V]
  TX Bias:     7.426 mA     [Range 0 mA - 131 mA]
  TX Power:   -2.5743 dBm   [Range -40 dBm - +8.2 dBm]
               0.5528 mW    [Range 0 mW - 6.5535 mW]
  RX Power:   -3.8384 dBm   [Range -40 dBm - +8.2 dBm]
               0.4132 mW    [Range 0 mW - 6.5535 mW ]

Wed February 07, 2024 08:22 AM

Hi Chris,

nice one, especially part about RX/TX Power I was not aware of, so far I was aware only IBM support utilities efc_power/emfc_power which can provide RX/TX Power but can run under root only.

Unfortunately on our new P10 with 16Gbit card provide just zeros for RX/TX stats :( 

So most probably not going to add it into our lpar2rrd OS agent ...

fcstat -e fcs0

...

FC SFP Information
  Vendor Name: 
  Vendor OUI:  00000000
  Vendor PN:   
  Temperature: 0.000 C     [Range -128 C - +128 C]
  Voltage:     0.000 V      [Range 0 V - +6.55 V]
  TX Bias:     0.000 mA     [Range 0 mA - 131 mA]
  TX Power:   0.0000 dBm   [Range -40 dBm - +8.2 dBm]
               0.0000 mW    [Range 0 mW - 6.5535 mW]
  RX Power:   0.0000 dBm   [Range -40 dBm - +8.2 dBm]
               0.0000 mW    [Range 0 mW - 6.5535 mW ]