As part of our continuous effort to simplify PowerVM management there were several enhancements delivered in HMC V9 R2 M950. You can refer to https://ibm.biz/BdffSr for more information on the enhancements done in HMC version V9.2.950.0.
HMC V10 R1 M1010 adds below features to continue enhancing features delivered in HMC V9 R2 M950 and to address multiple request for enhancements.
- Prepare VIOS for Maintenance (RFE 112804)
- Validation for redundancy for the storage/network provided by VIOS to client partitions
- Switch path of redundant storage/network to initiate failover
- Rollback to original configuration on failure of prepare
- Audit various validation and prepare steps performed
- Report any failure seen during prepare
- Change High Availability Mode of Shared Ethernet Adapter of VIOS
- Scheduled operations to automate the VIOS Config backups in HMC (RFE 149518)
- Ability to specify labels for FC ports, requires VIOS 3.1.2.0 or later (RFE 132094)
- Option to backup/restore Shared Storage Pool config in HMC
- Options to import/export the backup files to external storage
- HMC CLI options to backup/manage/restore VIOS Config/SSP Config (RFE 149518). mkviosbk, chviosbk, cpviosbk, lsviosbk, rstviosbk, rmviosbk commands have been added to support VIOS backup and restore.
- Option to fail over all virtual NICs from one VIOS to another (RFE 132175)
Prepare VIOS for Maintenance:
In HMC V9.2.950.0, an option to Validate Maintenance Readiness is provided for the running VIOS partition with active RMC connection. HMC users with all partitions and VIOSs access will be able to perform this operation. Extension to this function is to prepare the VIOS for Maintenance by gracefully switching the path of the storage/network provided to the client partition.
Select the Virtual I/O Server and click Actions > Validate Maintenance Readiness and Prepare task to open validate Maintenance Readiness window and option to prepare the VIOS for maintenance as in below image.
This task performs the validation of redundancy for storage and network provided by VIOS to the client partition and shows the impacted partitions not having redundancy. Audit section is newly added in this utility which will provide the detailed information on various steps being performed while validating the redundancy.
After redundancy is validated, user can prepare the VIOS for Maintenance by clicking “Prepare for Maintenance” button.
Prepare also performs validation steps. If there are errors or warnings during validation step and the check-box in the confirmation window was not selected then prepare operation will not be attempted. If there are no errors or warnings during validation, the operation is attempted by performing the failover operation. If errors occur during the failover operation, the HMC performs roll-back to revert the VIOS into its original configuration. And if the check-box is selected, then prepare will be continued even if there are errors/warnings during validation steps or error during failover operation. i.e. roll-back will not be attempted when the check-box is selected. During prepare steps, no other configuration changes on the systems or partitions should be attempted. On completion of the prepare, report will be shown as in below image.
HMC completes the following steps during prepare:
- Unconfigures the Virtual SCSI Target Device of the redundant virtual SCSI mappings. Unconfigure is done only when the storage mapped to the vHost is having redundant connection to the client partition from other Virtual IO Server.
- Unconfigures the Virtual Fibre Channel Server Adapter of the Virtual FC Mappings. Unconfigure is done only when the storage provided from the vFCHosts are visible to the client partition from other VIOS.
- Switches the Virtual NIC backing device to a redundant operational backing device from the other VIOS.
- Change the High Availability mode of the Shared Ethernet Adapter of the failover Network Bridge to "Standby" state so that all the network traffic flows from the redundant Shared Ethernet Adapter of the other VIOS. Once the VIOS maintenance activity is complete, User has to change the HA mode value back to desired setting.
After successful prepare for maintenance operation, user can view the audit report of various operations that are performed during the prepare in the Audit Trails pane. The Validation pane displays the completion message of the prepare maintenance operation that includes errors or warnings as in above image.
Change High Availability Mode of Shared Ethernet Adapter of VIOS
System -> Virtual Networks shows all the network bridges associated with the system. Which could have failover, load balancing enabled Shared Ethernet Adapters(SEA) combination as in below screen.
View network bridge displays the details of the Network Bridge including SEA details. Additional column is provided which displays the High Availability Mode(HA Mode) of the SEA.
Modify network bridge provides an option for the user to change the HA Mode of the SEA to one of : Auto, Disabled, Sharing or Standby. During prepare VIOS for maintenance, HA Mode of the SEA on the target VIOS will be made to Standby state to ensure that the network traffic is flowing from the other SEA from redundant VIOS. Once the VIOS Maintenance is complete, user can use this task to change the HA Mode to desired value.
Scheduled operations to automate the VIOS Config backups in HMC
Scheduled operation on the Virtual IO Server now provides an option to take the IO Configuration Backup. The task launch gives the options as in below images.
So, as in above images, user can select the date and time at which the IO Configuration Backup to be created for the VIOS. The Repeat tab provides an option to specify the interval and the days on which the Backup To be created. User can specify the name with which the Backup to be created in the “Options” tab. HMC overwrites the existing backup file with the new backup file. Selecting “Save Multiple Backups” will ensure that the backups are created with the EPOCH timestamp(UTC) appended to the specified backup file name. HMC will save maximum 5 backup files after which the oldest backup file will be deleted to save the new backup file for the specific VIOS.
Ability to specify labels for FC ports
HMC now provides an option to specify a label to the Fibre Channel Port (FC Port) associated with the VIOS.
Virtual Storage section of the VIOS has Fibre Channel Ports tab, which lists details of all the FC Ports associated with the VIOS which now contains the details of the “label” associated with the FC Port. 
User can rename the FC Ports with the desired names so that they can uniquely identify the FC Port while creating the mapping. Such as based on the storage which the FC port is created, user can name them as "DevPort", "TestPort" etc. FC Port label is supported in VIOS with version 3.1.2.00 and later.
Partition’s Virtual Storage display the Virtual Fibre Channel storage associated with it. The table display the label associated with the mapped FC Port of the VIOS.
While adding the new FC Port mapping or editing the connection of the FC Port mapping, label of the port is displayed so that user can uniquely identify the FC Port. 
Option to backup/restore Shared Storage Pool config in HMC
HMC V9.2.950.0 version provides an interface to backup IO Configuration of the VIOS. This function has been extended to backup Shared Storage Pool (SSP) backup of the VIOS. Options are provided to manage the SSP configuration backup taken for the VIOSs and also restore it.
To perform this task, you must have access to Manage VIOS images files task. To launch the task click on HMC Management -> Templates and OS images -> Manage Virtual I/O Server backups window as in below image.
The panel lists the SSP backups along with Virtual I/O Server Configuration Backup and Virtual I/O Server Backup sections as shown below.
The Shared Storage Pool Cluster Backup section displays the Backup Name, Managed System, Virtual I/O Server for which the backup was created, and Backup created time. If the Virtual I/O Server is not managed by an HMC, the Universally Unique Identifier (UUID) of the Virtual I/O Server is displayed. If the managed system is no longer managed by HMC, then Machine Type and Model System (MTMS) of the managed system is displayed.
Click "Backup SSP Cluster Configuration" to backup SSP configuration settings of the selected Virtual I/O Server. The Virtual I/O Server must be in running state with an active Resource Monitoring Control (RMC) connection to create the backup. Select the Managed system and the Virtual I/O Server from the drop-down list and enter the Backup File Name and click Apply to backup SSP configuration settings of the selected Virtual I/O Server. Fill in the info in the panel as below.
A message will indicate that HMC will backup SSP configuration, and stores that in HMC’s file system. User can check the status of the operation in task log.
The task log has an entry on the status of the operation.
If the VIOS is not part of SSP Cluster, then the backup will fail. The backup gets saved with <SpecifiedBackupName>.<ClusterName>.tar.gz format.
HMC internally executes VIOS command : “viosbr -backup -clustername <ClusterName> -file <FileName>” and after the command is successful, backup file is copied from VIOS and stored in the HMCs file system.
Click Refresh after the SSP configuration backup operation is successful to view the newly created backup that is listed under the Shared Storage Pool Cluster Backup section.
Please refer to : https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_viosbr.htm for more information about the viosbr command.
User can rename the existing backup file. To do that, select the Shared Storage Pool Cluster Backup that is saved in the HMC and select Action > Rename and enter the Backup File Name and click OK to rename the backup of the selected Virtual I/O Server configuration. Provide the proper name as new backup file name. The task log also will have entry of operation being performed.
User can remove the old backup file to free up the HMC’s disk space or remove stale backup files. Select one or more SSP backups that are saved in the Hardware Management Console (HMC) and select Action > Remove and click OK in the confirmation message window to remove the selected SSP backups associated with the VIOS from HMC.
To restore SSP to a previous configuration taken as backup, select the virtual SSP backup file that is saved in the HMC and select Action -> Restore to restore the SSP configuration settings with the selected configuration. A confirmation message is displayed after the operation is complete. Optionally, you can select the check-box in the confirmation window to restart the Virtual I/O Server and click OK to complete the restore operation successfully. When this checkbox is selected, and if restore needs to reboot of the VIOS, HMC reboots and restore the configuration again. If you do not select the check-box while restoring the configuration, HMC restores the configuration settings, but if the restore operation fails and the VIOS must be restarted, you must manually restart the VIOS and apply the same configuration settings again.
You can check the status of the restore I/O configuration operation in the Tasks window.
The task log has an entry on the status of the operation.
For restore operation, HMC copies the backup file to VIOS and executes the VIOS command : “viosbr -restore -file <FileName> -clustername <ClusterName> -curnode -type rsct”. The restore operation is enabled only if the targeted Virtual I/O Server is in an active state with a RMC connection.
Options to import/export the backup files to external storage
For all type of backups, there is an option “Import” provided to import the backup files from external storage to the HMC for the specific target VIOS. HMC supports NFS and SFTP as external storage servers.
To import VIOS IO Config Backup files from the NFS server click on “Import” button as in below screen
Select the target system and VIOS information from the dropdown. Enter comma separated list of valid VIOS IO Config Backup files to be imported in “Backup Files” section. And for remote server as “NFS Server”, enter details of the Remote NFS Server, Mount Location, Remote Directory, and Other Mount Options details and click Apply to import the backup files from the specified remote system.
Import can take longer time based on number of files getting imported. The task log has an entry on the status of the operation. If there is insufficient space in HMC, disk space then import operation might fail.
Import option is same for all the types of backup files.
HMC has limited disk space to store the backup files. User can clear the disk space by removing the old or stale backup files. Or they can export the backup files to external NFS or SFTP storage server and clear it from HMC. Export option is available for all types of backup files. User can select one or more backup files and export them.
To export VIOS Backup files to the SFTP server, select the backup files from the table, and select action on “Export” as in below screen.
To export it to SFTP server select SFTP server as remote server type, select the Authentication Type(either Password or Key) and enter details of Remote Server, User ID, Password or Key, and Remote Directory details and click Apply to export the backup file to the specified remote system.
Export can take longer time based on number of files getting exported. The task log has an entry on the status of the operation. After the export is completed, user can select backup files and delete them to clear the disk space.
HMC CLI options to backup/manage/restore VIOS Config/SSP Config
Below are the new commands and options newly added to the HMC release to support the various type of VIOS backups
- lsviosbk :
- List various VIOS backup stored in HMC.
- Filter option to get granular information.
- mkviosbk :
- Create VIOS IO Config/SSP config backup.
- Create VIOS backup with various config options.
- chviosbk :
- Rename VIOS backup file from HMC.
- cpviosbk :
- Import/Export VIOS backups to external NFS/SFTP server.
- rstviosbk :
- Restore VIOS IO Config/SSP config.
- Option to reboot the VIOS if required.
- chhwres with –so
- Switch over all virtual NIC backing devices on VIOS partition to other VIOS partitions
In HMC V10.1.1010, new option has been provided to failover all the vNIC backing devices on one VIOS to redundant backing device from other VIOS. If the vNIC does not have active backing device from any other VIOSs no action will be
taken on such backing device, and error will be shown to the user.
Below is the example of performing failover of active vNIC backing devices from vios2 to other operational backing devices from other VIOSs.
After failover, you can see that the backing devices from vios1 became active.
Below examples failover of active backing devices from vios1 to other operational backing devices from other VIOS. If the operation fails for the specific backing device failover, then the error will be shown with the details.
REST Jobs
/rest/api/uom/VirtualIOServer/$UUID/do/ValidateMaintenanceReadiness
/rest/api/uom/VirtualIOServer/$UUID/do/PrepareMaintenance
/rest/api/uom/ManagementConsole/$UUID/do/ListAvailableBackup
/rest/api/uom/ManagementConsole/$UUID/do/ManageVIOSBackup
/rest/api/uom/ManagementConsole/$UUID/do/ImportExportVIOSBackups
/rest/api/uom/VirtualIOServer/$UUID/do/BackupSSPConfig
/rest/api/uom/VirtualIOServer/$UUID/do/RestoreSSPConfig
/rest/api/uom/VirtualIOServer/$UUID/do/FailoverVNICBackingDevices
References :
https://ibm.biz/BdffSr
https://www.ibm.com/docs/en/power10/000V-HMC?topic=apis-jobs
https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_viosbr.htm
https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_backupios.htm
https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_installios.htm
https://www.ibm.com/support/knowledgecenter/POWER8/p8hb1/p8hb1_vios_backup_backup.htm