DLPAR Performance Improvement in PowerVM 2.2.5

By Pete Heyrman posted Tue June 09, 2020 09:32 AM

PowerVM 2.2.5 DLPAR

Dynamic logical partition (DLPAR) operations allow you to add or remove resources (logical processors, processor units, memory, I/O devices and such) to/from an active virtual machine (LPAR) without requiring a reboot of the partitions or restarting workloads.  Customers have found the DLPAR function useful in many different situations such as dynamic changes in workload requirements, disaster recovery and so on. DLPAR has been an integral part of the PowerVM offering for many years and most operations are able to be performed relatively quickly.  The exception to the quick response time is DLPAR of memory.  To ensure isolation of information between partitions, the request to change the amount of memory assigned to a partition can be quite time consuming.  With PowerVM 2.2.5, changes have been made in the PowerVM hypervisor that have significantly reduced the time required to make changes to the memory configuration of a partition.  The following sections will provide insight into these improvements along with some performance results.

DLPAR add of memory to partition

When the hypervisor adds memory to a partition, memory is cleared to ensure that the data from one partition cannot be passed to a different partition (partition isolation).  Memory is managed by the hypervisor in blocks that are called Logical Memory Blocks (LMB).  By default the LMB size is 256MB but can be set on a system wide basis to as small as 16MB.  Prior to PowerVM 2.2.5, when the operating system requested that an LMB be added to a partition, the hypervisor would synchronously clear the memory.  Most operating systems process these LMB requests one LMB at a time so the clear was not able to occur until the operating system added the LMB to the partition.  This single threaded processing added to the time required for the DLPAR operation.  Starting with PowerVM 2.2.5, when the HMC receives the request to add memory to a partition, the hypervisor is the first subsystem notified of this request.  Once the hypervisor has identified the specific LMBs to add to the partition, the hypervisor starts an asynchronous clear request of all the LMBs to be added.  Multiple threads will operate on the LMBs that are being added to reduce the time required to clear the memory.  By the time the operating system is able to request to add the LMBs, there is a good chance the memory is already cleared.  This overlapping and multiprocessing of the clear requests has resulted in up to a 70% reduction in the time to add memory to a partition.

DLPAR removal of memory from a partition

When memory is assigned to a partition, the memory is used for a variety of purposes such as application code, application data, operating system buffers, I/O buffers and so on.  If a request is made to remove some memory from the partition, there is a process that must be followed to ensure there is no loss of data and no security exposures.  Any changed data must be migrated to a different memory location or written to external storage.  I/O devices and other buffers that are used to communicate outside of the partition, must be reset to point to new memory buffers since the old buffers could later be assigned to a different partition.  To ensure partition isolation, the PowerVM hypervisor must re-check that the operating system has removed all partition references to the memory.  Prior to PowerVM 2.2.5 when the operating system requested to remove an LMB from the partition, the hypervisor scanned a long list of external references.  If an external reference is left by the operating system, the hypervisor will fail the request by the operating system to release the memory to provide security.  In PowerVM 2.2.5, in addition to the list of external references, the hypervisor is keeping a reference counter that tracks how many external references exist on an LMB basis.  This way, without doing any long running scanning, the hypervisor can track if there are any remaining references.  This change in design has significantly decreased the amount of time to remove memory from a partition.  IBM has observed elapsed times that are 3 to 200 times faster with the latest firmware when removing memory from the partition.

IBMi Customers

If you are running the IBMi operating system, there are two IBMi PTFs that provide additional performance improvements when performing dynamic lpar memory operations.  These PTFs are:
- IBMi 7.2 MF62858
- IBMi 7.3 MF62859
- No PTFs are required with IBMi 7.4 and later releases

Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions