AIX Virtual Fibre Channel Enhancement
In this post I want to discuss two topics. First, I want to tell you all about a new AIX 7.3 command called vfcstat. Second, I want to show some of the (not so recent) enhancements to the fcstat command.
New vfcstat command
A new Virtual Fibre Channel (VFC) client adapter enhancement was introduced with the release of AIX 7.3 Technology Level 2 Service Pack 1 (November 2023). A new command, called vfcstat, was introduced. This allows administrators to access information for VFC adapters, such as statistics, status, and activities.
Prior to this command, administrators could use the cat command to view the information stored in the /proc file system (in /proc/sys/adapter/fc/fcsX). The vfcstat command makes it easier for admins to view this information (without needing to know the location of the various files in /proc).
This command is available for AIX 7.3 systems that are Virtual I/O (VIO) clients, of VIOS running version 3.1.4.x and later (VIOS 3.1.4 provides support for NPIV multi-queue, https://www.ibm.com/support/knowledgecenter/POWER9/p9hb1/p9hb1_npiv_multi_queue_support.htm).
From the AIX 7.3 command reference for vfcstat:
“The vfcstat command reports information such as virtual FC adapter state, status, NPort ID, I/O queues information, read/write statistics, binding information with the host (VIOS), link errors, path timeouts, and so on. The required information is fetched in real time and displayed in the output. You can reset some of the details such as statistics and error counts. You can also retrieve read/write statistics in formatted output for scripting purposes by using the -F option. The vfcstat command displays the statistics in a specified time interval for the specific number of times. The vfcstat command works on virtual FC client adapter only.”
The new command is delivered with the devices.vdevice.IBM.vfc-client.rte fileset.
# lslpp -w /usr/bin/vfcstat
File Fileset Type
----------------------------------------------------------------------------
/usr/bin/vfcstat
devices.vdevice.IBM.vfc-client.rte Symlink
# ls -ltr /usr/bin/vfcstat
lrwxrwxrwx 1 bin bin 17 Oct 23 18:23 /usr/bin/vfcstat -> /usr/sbin/vfcstat
# lslpp -w /usr/sbin/vfcstat
File Fileset Type
----------------------------------------------------------------------------
/usr/sbin/vfcstat
devices.vdevice.IBM.vfc-client.rte File
# lslpp -lc devices.vdevice.IBM.vfc-client.rte
#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked
/usr/lib/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:
/etc/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:
I tested the vfcstat command on an AIX system running 7300-02-01-2346.
You can use the command to list all the VFC adapters on your AIX system.
# vfcstat -l
fcs0
fcs1
To list all the functions available for the given VFC adapter, run the following:
# vfcstat -d fcs0
activity
capabilities
hostinfo
link
nport_id
status
tunables
wwpn
To display error counts for a given VFC adapter.
# vfcstat -d fcs0 -e
Current time : Jan 22 2024, 22:00:52.294
Time since last reset : 5885149 sec ( 2 months 8 days 2 hours 45 minutes 49 seconds )
Link Errors:
Total errors: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path Timeouts:
Domain: Admin
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path Timeouts:
Domain: SCSI
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path Timeouts:
Domain: NVMeOF
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Note there are three domains for VFC adapter elements. One is for VFC Admin commands, the second is for SCSI commands and the third is for NVME commands.
Use the -s flag to display various statistics for a given VFC adapter. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/stats. We’ve included the –F option as it displays formatted statistic output (which is easier to read!).
# vfcstat -d fcs0 -s -F
Current time=Jan 22 2024, 22:38:58.102
Time since last reset=5887435 sec ( 2 months 8 days 3 hours 23 minutes 55 seconds )
Queue_Start
Domain=SCSI
Queue number=0
Read reqs=9814
Read bytes=196898541
Write reqs=2696118
Write bytes=253070213192
No DMA res count=0
No SGE count=0
h_send_dropped=0
h_send_closed=0
h_send_busy=0
No cmd res count=0
Queue_End
To display tunables of a given VFC adapter, use the -d and -f tunables options. This is same as reading proc file /proc/sys/adapter/fc/fcs0/tunables.
# vfcstat -d fcs0 -f tunables
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem ( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues ( Requested / Granted ): 4 / 0
To display VIOS host information of a given VFC adapter, use the -f hostinfo option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/hostinfo.
# vfcstat -d fcs0 -f hostinfo
VFC client adapter name : fcs0
Host partition name (VIOS) : vios1
VFC host adapter name : vfchost0
VFC host adapter location code : U8286.42A.2153B1V-V1-C30
FC adapter name on VIOS : fcs0
FC adapter location code on VIOS : U78C9.001.WZS00XE-P1-C1-T1
To display WWPN information of a given VFC adapter, use the -f wwpn option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/wwpn.
# vfcstat -d fcs0 -f wwpn
0xc852100000000011
To reset all fields in the stats function for a given VFC adapter.
# vfcstat -d fcs0 -s –z
To reset all error counts for a given VFC adapter.
# vfcstat -d fcs0 -e –z
Here's a little script I wrote to collect and display output from each of the available functions for all VFC adapters on a system:
|
myvfcs=$(/usr/bin/vfcstat -l)
vfcsfuncs=$(echo $myvfcs | awk '{print $1}' | xargs /usr/bin/vfcstat -d)
echo Displaying functions that are available for a virtual FC client adapter: $myvfcs
echo
echo Functions available for $myvfcs are:
echo $vfcsfuncs
echo
echo
for xvfcs in $myvfcs
do
for eachvfunc in $vfcsfuncs
do
echo "VFCSTAT $eachvfunc for $xvfcs" | xargs /usr/local/bin/box
/usr/bin/vfcstat -d $xvfcs -f $eachvfunc
echo
done
done
|
The output is:
# ./xvfcs
Displaying functions that are available for a virtual FC client adapter: fcs0 fcs1
Functions available for fcs0 fcs1 are:
activity capabilities hostinfo link nport_id status tunables wwpn
#############################
# VFCSTAT activity for fcs0 #
#############################
Admin Queue
Admin I/O Queue
#################################
# VFCSTAT capabilities for fcs0 #
#################################
Requested capabilities : CAN_MIGRATE, CAN_HANDLE_FPIN
Host responded capabilities :
#############################
# VFCSTAT hostinfo for fcs0 #
#############################
VFC client adapter name : fcs0
Host partition name (VIOS) : vios1
VFC host adapter name : vfchost0
VFC host adapter location code : U8286.42A.2153B1V-V1-C30
FC adapter name on VIOS : fcs0
FC adapter location code on VIOS : U78C9.001.WZS00YD-P1-C7-T1
#########################
# VFCSTAT link for fcs0 #
#########################
Link UP
topology FABRIC
speed 8Gb
#############################
# VFCSTAT nport_id for fcs0 #
#############################
0xe016a0
###########################
# VFCSTAT status for fcs0 #
###########################
State 0x10 (ONLINE)
Opened true
cfg_opn_state 0x0
prev_cfg_channels 0
flags 0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size 0x20000000
total_dma_size_used 0x9900000
no_chs_reason_code 0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count 0
online_time 373 (lbolt)
UP time 5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )
Admin Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
resp_cmds 0
Admin I/O Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
waiting_cmds 0
resp_cmds 0
offlevel_resp_cmds 0
cancel_resp_cmds 0
#############################
# VFCSTAT tunables for fcs0 #
#############################
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem ( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues ( Requested / Granted ): 4 / 0
#########################
# VFCSTAT wwpn for fcs0 #
#########################
0xc852100000000011
#############################
# VFCSTAT activity for fcs1 #
#############################
Admin Queue
Admin I/O Queue
#################################
# VFCSTAT capabilities for fcs1 #
#################################
Requested capabilities : CAN_MIGRATE, CAN_HANDLE_FPIN
Host responded capabilities :
#############################
# VFCSTAT hostinfo for fcs1 #
#############################
VFC client adapter name : fcs1
Host partition name (VIOS) : vios2
VFC host adapter name : vfchost0
VFC host adapter location code : U8286.42A.2153B1V-V2-C30
FC adapter name on VIOS : fcs0
FC adapter location code on VIOS : U78C9.001.WZS00XE-P1-C1-T1
#########################
# VFCSTAT link for fcs1 #
#########################
Link UP
topology FABRIC
speed 8Gb
#############################
# VFCSTAT nport_id for fcs1 #
#############################
0xe016c0
###########################
# VFCSTAT status for fcs1 #
###########################
State 0x10 (ONLINE)
Opened true
cfg_opn_state 0x0
prev_cfg_channels 0
flags 0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size 0x20000000
total_dma_size_used 0x9900000
no_chs_reason_code 0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count 0
online_time 373 (lbolt)
UP time 5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )
Admin Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
resp_cmds 0
Admin I/O Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
waiting_cmds 0
resp_cmds 0
offlevel_resp_cmds 0
cancel_resp_cmds 0
#############################
# VFCSTAT tunables for fcs1 #
#############################
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem ( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues ( Requested / Granted ): 4 / 0
#########################
# VFCSTAT wwpn for fcs1 #
#########################
0xc852100000000013
Enhancements to fcstat
In a somewhat related set of enhancements, the fcstat command was also updated with several new options that provide additional information to the administrator. You may or may not be aware, but these new options have been available for some time with AIX 7.3. The new options are:
· -p Displays a time-series report of traffic statistics for a specific transport protocol (TP) that is specified with the Protocol parameter.
· -t Displays a time-series report of the traffic statistics continuously with a time interval between two consecutive reports, which is specified with the Interval parameter.
· -e Displays all the statistics, that includes the device-specific statistics such as driver statistics, link statistics, FC4 types, and vital Small Form-factor Pluggable (SFP) parameters.
· -d Displays the statistics by opening the adapter in diagnostic mode.
Here’s an example of using the -t and -p options. In the example below we have chosen to view the SCSI protocol statistics for the fcs0 adapter and to display the time-series report of traffic statistics with a one second interval.
# fcstat -t 1 -p scsi fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: Virtual Fibre Channel Client Adapter (adapter/vdevice/IBM,vfc-client)
TP Read Reqs(K) Write Reqs(K) Read (GB) Write (GB)
===== ================= ================= =========== ===========
SCSI 9 2314 0 203
SCSI 9 2315 0 203
SCSI 9 2316 0 203
SCSI 9 2317 0 203
SCSI 9 2318 0 203
SCSI 9 2319 0 203
SCSI 9 2320 0 203
SCSI 9 2322 0 204
SCSI 9 2323 0 204
SCSI 9 2324 0 204
SCSI 9 2326 0 204
SCSI 9 2328 0 204
SCSI 9 2329 0 205
SCSI 9 2330 0 205
SCSI 9 2332 0 205
SCSI 9 2334 0 205
SCSI 9 2335 0 205
SCSI 9 2336 0 206
SCSI 9 2338 0 206
SCSI 9 2340 0 206
SCSI 9 2341 0 206
SCSI 9 2342 0 206
SCSI 9 2344 0 207
SCSI 9 2346 0 207
SCSI 9 2347 0 207
SCSI 9 2348 0 207
SCSI 9 2350 0 207
SCSI 9 2352 0 208
SCSI 9 2353 0 208
SCSI 9 2354 0 208
In the output above the time-series report shows the following statistics:
· Read Reqs(K): The number of read or input requests, in thousands.
· Write Reqs(K): The number of write or output requests, in thousands.
· Read(GB): The number of input bytes that are read, in gigabytes.
· Write(GB): The number of output bytes that are written, in gigabytes.
If you do not have any NVME adapters/devices on your AIX system (or you are only interested in FC SCSI traffic), you can remove the -p flag and shorten the command to, fcstat -t 1 fcs0.
The -e option displays vital Small Form-factor Pluggable (SFP) parameters such as vendor information, temperature, voltage information, as well as, the device-specific statistics such as driver statistics, link statistics, and FC4 protocol types.
Previously, some of this information was only available using a special tool (called emfc_power) provided by IBM support. This option is useful with physical (not virtual) fibre channel adapters. It appears this enhancement may have been delivered based on this customer enhancement request, https://ibm-power-systems.ideas.ibm.com/ideas/AIX-I-688.
# fcstat -e fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: PCIe3 2-Port 32Gb FC Adapter (df1000e314101506) (adapter/pciex/df1000e31410150)
...etc...
FC SFP Information
Vendor Name: AVAGO
Vendor OUI: 0000176a
Vendor PN: AFBR-57G5MZ-ELX
Temperature: 42.371 C [Range -128 C - +128 C]
Voltage: 3.331 V [Range 0 V - +6.55 V]
TX Bias: 5.466 mA [Range 0 mA - 131 mA]
TX Power: -1.4837 dBm [Range -40 dBm - +8.2 dBm]
0.7106 mW [Range 0 mW - 6.5535 mW]
RX Power: -3.6866 dBm [Range -40 dBm - +8.2 dBm]
0.4279 mW [Range 0 mW - 6.5535 mW ]
The fcstat command opens the device in diagnostic mode by using the -d flag, as shown below. When the link is down and the device is opened in nondiagnostic mode, the fcstat command delays in generating the output. You can use the -c flag to remove this delay. If the device is already opened and the fcstat command is started with the -d flag, the open operation on the device fails with an EACCESS error. Meaning you can now run fcstat on an adapter, with a link that is down, and still obtain some useful information relating to link status, failure, and loss of sync/signal counts.
# fcstat -d fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03) (adapter/pciex/df1000f114108a0)
Serial Number: 1A202029F7
Option ROM Version: 0278213D
ZA: U2D2.10X13
World Wide Node Name: 0x20000000C9D55D1E
World Wide Port Name: 0x10000000C9D55D1E
FC-4 TYPES:
Supported: 0x0000012000000000000000000000000000000000000000000000000000000000
Active: 0x0000010000000000000000000000000000000000000000000000000000000000
Class of Service: 3
Port Speed (supported): 8 GBIT
Port Speed (running): 0 GBIT
Port FC ID: 0x000000
Port Type: UNKNOWN
Attention Type: Link Down
Topology: UNKNOWN
Seconds Since Last Reset: 0
Transmit Statistics Receive Statistics
------------------- ------------------
Frames: 0 0
Words: 0 0
LIP Count: 0
NOS Count: 0
Error Frames: 0
Dumped Frames: 0
Link Failure Count: 4
Loss of Sync Count: 4
Loss of Signal: 5
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
FC SCSI Adapter Driver Information
No DMA Resource Count: 0
No Adapter Elements Count: 0
No Command Resource Count: 0
FC SCSI Traffic Statistics
Input Requests: 0
Output Requests: 0
Control Requests: 0
Input Bytes: 0
Output Bytes: 0
You can read more about these new options for fcstat in this blog from Yadagiri Rajaboina from IBM AIX Storage Drivers Development.
Displaying the Fibre channel HBA SFP information with AIX/VIOS
https://community.ibm.com/community/user/power/blogs/yadagiri-rajaboina/2023/10/29/displaying-fc-hba-sfp-information?CommunityKey=daa942cb-b783-4fd3-ba27-a2d7462f9530
I encourage you to read the command reference guides, and also the “What’s new” links below, to learn more about the latest enhancements.
References:
vfcstat Command Reference
https://www.ibm.com/docs/en/aix/7.3?topic=v-vfcstat-command
fcstat Command Reference
https://www.ibm.com/docs/en/aix/7.3?topic=f-fcstat-command
What's new in Commands - November 2023
https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__3
What's new in Commands - November 2022
https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__4
#power-featured-area-3