Power Virtualization

Power Virtualization

Learn about the virtualization technologies designed specifically for IBM Power including #PowerVM, #PowerVC, #VM Recovery Manager, #HCM/CMC, and more.


#Power
#TechXchangeConferenceLab

 View Only

Power8 PCIe Slot Concurrent Maintenance Overview

By Travis Pizel posted Thu June 18, 2020 08:44 AM

  
Introduction
Did you know you can replace, add, and remove many types of hardware on your POWER8 systems running PowerVM without powering off the system?  This is referred to as hardware concurrent maintenance.  IBM has supported concurrent maintenance on its Power line of computer systems for many years.  The capabilities have evolved over time and now include support for many hardware components.  Power supplies, fans, disk drives, PCIe adapters, and I/O expansion drawers are some of the more common ones. 

The ultimate goal of concurrent maintenance, sometimes referred to as 'hot plug,' is to minimize disruption to workloads when you must replace, add, or remove hardware components.  Concurrent replacement is most often done to replace a failed or failing part with a new one.  Concurrent add is typically done to add additional capacity to a system.  Concurrent remove is often used in conjunction with concurrent add to do any of a variety of useful things such as upgrade a particular component to a higher performance or higher capacity version, move an I/O component from one location in the system to another to better balance the I/O resources across the system configuration and improve performance, or migrate physical resources from one data center to another. 

It should be noted that not all systems in the Power8 family have the same concurrent maintenance capabilities.   Enterprise class systems have more concurrent maintenance capabilities than low end systems.  Similarly, systems managed by an HMC have more capabilities than systems not managed by an HMC. 


This article will focus on PCIe slot concurrent maintenance on Power8.  PCIe slot concurrent maintenance is supported on all PowerVM models in the Power8 family, and is a key Reliability, Availability, and Serviceability (RAS) differentiator between IBM and many of its competitors in the low end space.

How It's Done
PCIe slot concurrent maintenance on Power8 allows you to add PCIe adapters to, remove PCIe adapters from, and replace PCIe adapters in PCIe slots.  This is supported for all standard PCIe slots on all Power8 systems running PowerVM, including PCIe slots in a PCIe Gen3 I/O Expansion Drawer.

For a system that is not managed by an HMC, PCIe slot concurrent maintenance is initiated from the operating system running in the partition that owns the target slot.  In IBM i, it is launched by specifying the concurrent maintenance option against the Packaging Hardware Resource for the slot in the Hardware Service Manager utility, as shown in the following figure.   Hardware Service Manager is available in System Service Tools and Dedicated Service Tools.

PowerVM Concurrent Maintenance IBMi
In AIX and VIOS, it is launched via the PCI Hot Plug Manager task in the diag utility as shown in the following figure:
PowerVM Concurrent Maintenance AIX
In Linux, it is launched from the command line with the drmgr command as shown in the following figure:
PowerVM Concurrent Maintenance LinuxIf the system is managed by an HMC, PCIe slot concurrent maintenance can be launched from the operating system running in the partition that owns the target slot as discussed above, or it can be launched via the Exchange FRU, Add FRU, or Remove FRU task under Serviceability > Hardware, as shown in the following figure:
PowerVM Concurrent Maintenance HMCFrom the HMC, it is also possible to launch PCIe slot concurrent replace via the 'Repair' button in a serviceable event that calls out the PCIe slot location as a failing part. 

If the system is managed by an HMC, you may prefer doing PCIe slot concurrent maintenance from the HMC rather than the operating system of the partition that owns the slot.  The HMC automatically guides you through each step of the procedure, with detailed instructions and diagrams that are specific to the enclosure in which the target location resides.  It also activates and deactivates the identify indicator for the target location automatically, providing additional assistance locating the part and confidence that you are working with the correct location.

In cases where PCIe slot concurrent maintenance is done from the operating system of the partition that owns the target slot, instructions can be found in Knowledge Center.  For example, instructions for replacing a PCIe adapter in a PCIe Gen3 I/O Expansion Drawer can be found at https://www.ibm.com/support/knowledgecenter/P8ESS/p8egy/p8egy_emx0_pcie_adapter_rr_kickoff.htm.  The first page is shown in the following figure:

PowerVM Concurrent Maintenance Knowledge Center

Resource Deconfiguration
If you do concurrent maintenance on a slot that is owned by an active partition, you may be asked to deconfigure operating system specific logical resources associated with the adapter in the slot.  In many cases, deconfiguration of these resources is required before the utility will allow you to complete the concurrent maintenance procedure. 

The concurrent maintenance utilities in HMC, IBM i, AIX, and VIOS block the concurrent maintenance procedure until required resource deconfiguration has been completed, and provide the names of the resources that must be deconfigured.   This prevents the user from inadvertently powering off an adapter while it is being used by the operating system. 

Linux, on the other hand, does not provide this safeguard for most adapters.  It is the user's responsibility to determine whether there are workloads actively using an adapter owned by a partition running Linux before doing concurrent maintenance on that adapter from the HMC or the Linux operating system.

In IBM i, resources are deconfigured using the VRYCFG command from the XPF command line.  In AIX and VIOS, the rmdev command from the command line is used to deconfigure resources.

Minimizing Workload Disruption
When you deconfigure I/O resources and power off I/O adapters, those resources are no longer available for use by workloads running in the partition that owns the resources.  Access to associated I/O devices such as disk drives and network connections is lost.  Obviously, this can be disruptive to workloads running on the system.  This disruption can be minimized or eliminated completely if the system is configured with concurrent maintenance in mind. 

For disk or solid state drives, multi-path I/O should be employed.  Minimally, each path should go through a different I/O adapter.  If the adapters are in PCIe Gen3 I/O Expansion Drawers, if possible, each path should go through a different expansion drawer.  Disk mirroring can also be employed, with the same configuration rules as multi-path I/O.

For network connections, Network Interface Backup (NIB) should be employed such that each network connection in the backup set is on a different adapter.  If the adapters are in PCIe Gen3 I/O Expansion Drawers, each adapter should be in a different drawer if possible.

Redundant VIOS partitions provide a convenient way to do concurrent maintenance on I/O hardware without disrupting workloads.  Configure multi-path I/O in the client partitions for the disk resources, where each path goes through a different VIOS.   Configure Shared Ethernet Adapters (SEA) in the VIOS partitions such that there are pairs of SEA adapters providing access to the external network for client partitions with each SEA in a pair configured in a different VIOS.   Virtualize all I/O for the client partitions through the redundant VIOS partitions such that backing devices for the virtual I/O can be accessed through two or more VIOS partitions, and each VIOS accesses a given backing device through a different PCIe adapter.  If the adapters are in PCIe Gen3 I/O Expansion Drawers, each VIOS should access a given backing device through a different expansion drawer if possible. Then, when you must do concurrent maintenance on an adapter owned by a VIOS, you can either deconfigure the associated resources in that VIOS or power off the entire VIOS (much simpler) without causing client partitions to lose access to I/O and disrupting workloads in those partitions. 

Limitations and Restrictions
There are some restrictions on PCIe slot concurrent maintenance involving Coherent Accelerator Processor Interface (CAPI) adapters and PCIe cable adapters, both of which are new for Power8.  Specifically, concurrent remove is not supported for either, and concurrent add of PCIe cable adapters is not supported on the low end 8284 and 8286 systems. 

Additional Information

POWER8 Processor-Based Systems RAS whitepaper

IBM PowerVM Virtualization Introduction and Configuration redbook

Contacting the PowerVM Team
Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions







#PowerVM
#PowerVM
#powervmblog
0 comments
11 views

Permalink

https://community.ibm.com/community/user/blogs/travis-pizel1/2020/06/18/power8-pcie-slot-concurrent-maintenance-overview