Reduce Firmware Memory by Limiting the Number of External Drawers
Introduction
The Power Hypervisor pre-allocates I/O translation table space for each PCI-E slot in the system. With the introduction of the NED24 and ENZ0 expansion drawers, the amount of memory reserved for cable card capable slots, intended to support dynamic addition of these drawers has increased significantly. As a result, a considerable portion of system memory is blocked from other uses.
To address this, FW1110 for Power11 (as well as FW1060 for Power10) servers allows customers to control how platform memory is utilized—either to support dynamic expansion drawer additions or to allocate the memory for logical partition (LPAR) use. This feature aims to optimize memory usage by reducing unnecessary pre-allocation of I/O translation table space and making it configurable for end users.
Specification of Slot Count
A new ASM attribute "Dynamic I/O Drawer Attachment Slot Count" provides customers a method to specify how many PCI-E slots to pre-allocate I/O translation table space to support dynamic addition of expansion drawers.
ASM for system models E1180/E1080 system
An ASMI attribute “Dynamic I/O Drawer Attachment Slot Count” under “System Configuration” menu.
ASM for system models E1150/ E1050
An ASMI attribute “Dynamic I/O Drawer Attachment Slot Count” under “Resource management” menu.
Based on the slot count selection, cable card capable slots that are not occupied by a cable card are pre-allocated I/O translation table space based on the defined priority order for the platform type. For more information on the priority order, reference the slot properties documentation for the appropriate platform type.
The Slot properties documentation for Power11 Systems:
https://www.ibm.com/docs/en/POWER11/p11eab/p11eab_1180_slot_details.htm
https://www.ibm.com/docs/en/POWER11/p11eab/p11eab_1150_slot_details.htm
https://www.ibm.com/docs/en/POWER11/p11eab/p11eab_1122_slot_descriptions.htm
https://www.ibm.com/docs/en/POWER11/p11eab/p11eab_1114_1124_slot_descriptions.htm
For Power10 systems:
https://www.ibm.com/docs/en/POWER10/p10eab/p10eab_1080_slot_details.htm
https://www.ibm.com/docs/en/POWER10/p10eab/p10eab_1050_slot_details.htm
https://www.ibm.com/docs/en/POWER10/p10eab/p10eab_1022_slot_descriptions.htm
https://www.ibm.com/docs/en/POWER10/p10eab/p10eab_1014_1024_slot_descriptions.htm
Note : The value can be modified only when the system is powered off.
Example:
Consider a 2 Node model E1180 system with no cable cards installed in the system with “Dynamic I/O Drawer Attachment Slot Count “ set to 0 for both nodes (ASM screen shot, left). With this setting, the reserved memory (memory used by the hypervisor) is about 74GB (Hardware Management Console screen shot, right).
With the "Dynamic I/O Drawer Attachment Slot Count" set to 8 for each node, the reserved memory for the same system increases to about 165GB, an increase of 91GB.
As shown, this new option enables the optimization of platform memory based on the user's requirements.
If a user attempts to add a drawer to a slot, at system runtime, that does not have pre-allocated I/O translation table space, the concurrent maintenance operation will fail during the "Power on power domain" step. The error message displayed will state: "The requested procedure cannot be completed because the type of card detected in the target slot is not supported by the target slot.
Configurable slot count based on the system type
The table below outlines the default and maximum values for Power11 and Power10 system models.
Power11 systems pre-allocate I/O translation space for only a subset of slots by default; customer action is required to enable additional slots for dynamic expansion support.
In contrast, for Power10 systems, the default behaviour is to pre-allocate I/O translation table space for all cable card-capable slots to support the dynamic addition of expansion drawers.
Power11 Systems:
|
Type
|
Default
|
Max
|
|
E1180
|
2
|
8
|
|
S1124, S1122, S1122, S1114, L1124, L1122
|
0
|
0
|
|
E1150
|
2
|
9
|
Power10 Systems:
|
Type
|
Default
|
Max
|
|
E1080
|
8
|
8
|
|
S1024, S1022, S1014
|
4
|
4
|
|
E1050
|
9
|
9
|
Summary
The technique described in this blog can significantly reduce the firmware memory that is reserved by setting a user defined limit on the number of I/O drawers that can be attached. If the need arises to attach I/O drawers beyond the specified limit, a scheduled reboot of the server will allow the limit to be increased.
Contacting the PowerVM Team
Have questions for the PowerVM team or want to learn more? Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions