About the authors

• Brian Preston (E-Mail: bpreston@us.ibm.com) - HMC Service Focal Point Team Lead
• HARIGANESH Muralidharan (Email: harimura@in.ibm.com) -STSM Systems Management Architect
• Osvaldo Rodriguez (Email: osvaldo@us.ibm.com) -STSM Custom Test Partnership
• Sougata Sarkar (Email: sougsark@in.ibm.com)-Custom Test Partnership and Cognitive Guild Lead 
• Thaitiel Valencia (Email: thaitiel@ibm.com) - Custom Test Partnership
• Hari Chamanchi (Email: chamanchi.hari@in.ibm.com)- Integrated Software Systems Test
• Juan Deanda (Email: jrdeanda@us.ibm.com)- HMC/Service Focal Point Development

This document will guide you through installing I/O device microcode from the Hardware Management Console (HMC).

I/O device microcode is a low-level firmware that governs the behavior and functionality of hardware adapters (such as Ethernet or Fiber Channel adapters). Keeping this microcode up to date is essential for:

• Security patching
• Enabling new features
• Applying bug fixes
• Improving performance
• Ensuring hardware compatibility

In this blog, walk you the process of installing I/O device microcode in IBM Power systems from the Hardware Management Console (HMC).

The traditional method of updating I/O device firmware is through VIOS.  This method involves several manual steps:

• Login to the VIOS
• Identify the adapter
• Note the Feature Code (FC) of the adapter
• Download the corresponding microcode from IBM Fix Central
• Use the diag utility to apply the update manually

While effective, this method is time-consuming, requires system-level access to each VIOS, and updates must be run from each VIOS.

Using the HMC is a more efficient approach. This method allows updates to be managed centrally or use the online repository hosted by IBM.

• Centralized Management: No need to log in to each VIOS individually
• Simplified Process: Reduces manual steps and potential for human error
• Faster Deployment: Especially beneficial in environments with multiple systems
• Better Audit and Control: HMC provides logs and status tracking for updates
• Supports Automation: Can be integrated into broader system update workflows

Before starting, there are some basic requirements that must be met.

• Devices being updated must be assigned to a partition. Updates are only supported for IO adapters assigned to VIOS, AIX, or Linux partitions
• Server must be in Operating state and partitions must be active
• RMC connections must be active on the HMC for the partitions to which IO adapters are assigned
• The catalog of I/O device firmware must be available, either online by using the “IBM Service Web Site” repository, or locally
• If using the “IBM Service Web Site” online repository from which to download firmware, reference the IBM Security Whitepaper at https://public.dhe.ibm.com/linux/service_agent_code/LINUX/HMC_SECURITY_WHITEPAPER.PDF regarding setting up outbound configuration and which IBM addresses and ports are used, specifically in the “Fix / Policy download Servers” section

Accessing the firmware microcode:

If you do not have an external internet connection, or wish to locally host the microcode, you can obtain the code in the form of an .iso file from IBM.  At the time of this writing, the .iso and instructions can be obtained at https://www.ibm.com/support/pages/node/6491185 with a file size of 1.5GB.  When unpacked, the code is required to follow a specific directory pattern on the media.  It is preformatted for this pattern, and ready to use.  However, if the files are copied off the media and hosted elsewhere, the same file and directory structure must be maintained for updates to work.

The easiest way to work with the .iso file is to mount it on a file system, which requires root access.  This can be done on a local NFS or SFTP server.  This will automatically preserve the required directory structure.  To do so, run the following steps on the remote hosting server:

1. Become root user
2. Copy the .iso file to the hosting server
3. Create a directory which will be the remote repository directory from the HMC, such as /ioMicrocode (“mkdir /ioMicrocode”)
4. Mount the .iso file to that directory (“mount -o loop /pathToIso/io_image_all.iso /ioMicrocode”)
5. Verify the mount if desired (“mount | grep io_image_all”)
6. List the contents if desired (“ls -lat /ioMicrocode”).  As you can see, the “/ioMicrocode /microcode” directory will contain all of the device image folders and the microcode catalog file.  Within each device image folder is the RPM that holds the device firmware to be installed.  Keep in mind that you will specific only up to the “/ioMicrocode” path from the HMC.  Do not add the “microcode” directory to the path from the HMC.  It is automatically done.

[root@remoteHostingServer /] # ls -lat /ioMicrocode

-rw-r--r--   1 591499 366170 431538 Feb 28 19:06  cddownload.html

-rw-r--r--   1 591499 366170 361359 Feb 28 18:44  cddownload.txt

drwxrwsr-x 354 591499 366170  45056 Feb 28 18:27  microcode

drwxrwsr-x   3 591499 366170   2048 Sep 20  2022  lib

drwxrwsr-x   2 591499 366170   2048 Sep 20  2022  invscout

drwxrwsr-x   2 591499 366170   2048 Sep 20  2022  info

-rw-r--r--   1 591499 366170  25692 Jan 11  2022 'ISO Image Readme.html'

-rw-r--r--   1 591499 366170  86120 Oct 20  2021 'License Agreement for Machine Code.html'

-rw-rw-r--   1 591499 366170   9816 Jan  7  2008  which_tool

-r-xr-xr-x   1 591499 366170   3274 Sep 21  2006  MDT

-rwxr-xr-x   1 591499 366170   1498 Apr  3  2003  slogoFFFFFF.gif

-rw-r--r--   1 591499 366170   1944 Jul  8  1999  impact.html

-rw-r--r--   1 591499 366170   4165 Feb  4  1999  severity.html

Examples of the content within the microcode and a specific device microcode directory are listed below, but they are just for reference and are not specified by the user.

[root@remoteHostingServer /] # ls -lat /ioMicrocode/microcode

-rwxr-xr-x   1 591499 366170 333120 Feb 28 17:58 catalog.mic

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:47 FESK1_A

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:46 FESKJ_A

drwxrwxr-x   2 591499 366170   2048 Feb 28 18:27 FEKF2_A

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:26 FES1E_A

drwxrwxr-x   2 591499 366170   2048 Feb 28 18:25 FEN2W_A

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:24 FESDF_A

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:24 FES3A_A

drwxrwsr-x   2 591499 366170   2048 Feb 28 18:22 FEN2A_A

...etc...

[root@remoteHostingServer /] # ls -lat /ioMicrocode/microcode/FESK1_A

total 13407

drwxrwsr-x   2 591499 366170    2048 Feb 28 18:47 .

drwxrwsr-x 354 591499 366170   45056 Feb 28 18:27 ..

-rwxr-xr-x   1 591499 366170 3523510 Sep  8  2022 nytro3732-4085-AIX.rpm

-rwxr-xr-x   1 591499 366170 3523507 Sep  8  2022 nytro3732-4085-LINUX.rpm

-rw-r--r--   1 591499 366170  134048 Sep  8  2022 Nytro3732.html

-rw-r--r--   1 591499 366170   11909 Sep  8  2022 Nytro3732.readme.txt

-rwxr-xr-x   1 591499 366170 2162688 Aug 31  2022 XS1600M.A1800022.34303835

-rwxr-xr-x   1 591499 366170 2162688 Aug 31  2022 XS400ME.A1800022.34303835

-rwxr-xr-x   1 591499 366170 2162688 Aug 31  2022 XS800ME.A1800022.34303835

7. In this example, the “/ioMicrocode” directory can now be used as the repository location from the HMC when using a remote location such as an FTP or NFS server.  The HMC will automatically search for the “microcode” folder, catalog file, and device image folders under the specified path.  You do not specify the “microcode” folder in the path of the repository.

If you do not wish to mount the .iso file directly, you may extract the contents on the remote hosting server using a program that supports that operation.  Note that the above-mentioned directory structure must be maintained.

NFS repository support is added in HMC Version 11.  Mount the ISO on an NFS server like the above instructions.  Give a remote HMC access to the NFS server by following instructions for setting up NFS server access.  For NFS servers, the NFS mount point will be specified as the “location” option from the HMC.  Any directory under that NFS mount point can be specified using the “directory” option.  The NFS mount point will be listed in the /etc/exports file on the NFS server along with the allowed HMC addresses.

It is also possible to host the unpackaged files on the local HMC.  Either extract the files from the iso, or after mounting on a remote server, transfer the files to the local HMC file system.  The above-mentioned directory structure must be maintained but the catalog.mic and microcode directory can be created under a parent directory.

Verifying RMC:
To determine if RMC communication is established between the HMC and partitions, run the "diagrmc" command from the HMC command line.  You can run "diagrmc" with no options to target every server, or run against a specific server by using the “-m” option, for example "diagrmc -m 8375-42A*0000000".  If RMC is not active, run "rmcctrl -s" to activate it.  RMC needs to be running on both the HMC and the partition and the command to start it will work on both platforms.  “diagrmc” requires superuser permission on the HMC and partition.

Example output when managed server is not powered on.  To fix this, power on the managed server.

diagrmc

** Checking management console network setup, NIST mode, firewall, RMCkey, etc.:

        -- System 8408-44E*00000000 is not in Operating state.

Example output when a partition does not have an active RMC connection.  To fix this, start RMC on the partition as per the command listed above.

diagrmc

** Checking management console network setup, NIST mode, firewall, RMCkey, etc.:

** Checking RMC subsystem:

        >> WARN: Running lpar partitionName (2*8408-44E*0000000) does not have an active RMC connection:

Updating devices:

Once the HMC and partitions have a valid RMC connection, and the devices to be updated are assigned to a partition, you can launch the HMC GUI to start the update or use the CLI “updlic” command.

1. Select the managed server from the Systems menu
2. Start the "Update IO Firmware" task

3. Accept the license and click “Next”

4. Select the desired I/O device microcode repository

If not using the “IBM Service Web Site” repository, the selected repository must follow the required directory structure to host the device firmware.  As mentioned above, the bundled .iso package is pre-formatted with this structure.  The head directory is “microcode” which contains the catalog file describing all available device firmware.  Under that are directories for each device.  Example: “microcode\FEN0W_A”.  Within each device directory is the firmware itself for each specific device.  Failure to maintain this structure will result in a failure of the update process. 

When choosing a repository location, select the directory that contains the microcode directory.  For example, if the local directory structure is “/ioMicrocode/microcode”, specify “/ioMicrocode” for the repository location.

5. Select which device(s) to update.  By default, nothing is selected.  Select the checkbox to indicate I/O device firmware should be.  Make note of the effect of updating a device.  Certain updates will take a device offline while the update is being performed, noted as “take offline” in the effect column.

NOTE: Concurrent I/O updates still result in a brief I/O outage of up to 40 seconds.

6. After selecting the devices to update, continue to the summary panel.  Click the “Install” button to proceed with the installation.

To use the command line to perform the update, use the “lslic” and “updlic” commands.  “lslic” is used to display the current firmware, or firmware available at a given repository.  “updlic” is used to perform an update against a specific device given the partition ID, and the device name, or all devices at once.  See the command help text or man pages for more detailed information on command usage.

Example “lslic” command for querying the current I/O device firmware levels:

lslic -t io -m 9105-41B*0000000

Example “lslic” commands for querying the current I/O device firmware levels and available updates from a repository:

lslic -t io -m 9105-41B*0000000 -r ibmwebsite

lslic -t io -m 9105-41B*0000000 -r sftp -h hostname -u user -d directory

lslic -t io -m 9105-41B*0000000 -r nfs -h hostname –location /nfsMountDirectoryOnHMC -d optionalSubDirectoryUnderMount

When querying the server to show current levels, no available level of effect will be shown, as no updates are being done.  Only the current level is shown.

# lslic -t io -m 9009-42G*0000000

partition=001,logical_device=ent0,device=PCIe3 2-port 100 GbE (NIC and RoCE) QSFP28 Adapter,mtms=9009-42G*0000000,current_level=001200201010,available_level=None,effect=None,reboot=None,suggested_action=None

When querying a repository for available updates, an available update and impact are shown if an update is found.

lslic -t io -m 9009-42G*13C6F8W -r mountpoint -d /media/dvd

partition=001,logical_device=ent0,device=PCIe3 2-port 100 GbE (NIC and RoCE) QSFP28 Adapter,mtms=9009-42G*0000000,current_level=001200201010,available_level=001200291017,effect=Reboot,reboot=Y,suggested_action=Update

Example “updlic” commands for updating a specific I/O device

updlic -m 9105-41B*0000000 -t io -o a -r nfs --location /nfsMountDirectoryOnHMC --id 1 --device nvme0 -v

updlic -m 9105-41B*0000000 -t io -o a -r nfs --location /nfsMountDirectoryOnHMC -d optionalSubDirectoryUnderMount --id 1 --device nvme0 -v

Example “updlic” command for updating all I/O devices

updlic -m 9105-41B*0000000 -t io -o a -r ibmwebsite -l latesat -v

updlic -m 9105-41B*0000000 -t io -o a -r ibmwebsite -l latesatconcurrent -v

There is no “level” argument to specify the new firmware package.  That is because the update process uses the inventory scout program to query the device and microcode catalog file in the repository to determine the latest available level for the device, searches the specified repository for that level, and installs it.

Summary

Using the HMC to perform I/O adapter microcode updates provides a centralized, efficient, and reliable method to keep hardware firmware current, reducing operational effort while improving consistency, security, and system stability.