View Only

Understanding the Benefits of Dark Cores and Memory

By Pete Heyrman posted Tue June 16, 2020 01:50 PM

PowerVM COD Logo
Dark or Unlicensed cores and memory are resources that are installed in your Power server but are not initially available for use by applications.  The following article provides some best practice information about how unlicensed resources can provide additional performance and reliability value to Power servers.

Unlicensed Memory
Unlicensed memory is a situation where a customer orders a system with more installed memory than they initially require with the expectation that over time the amount of required memory will grow as their business grows.  For entry class servers, generally all the installed memory is licensed for use but on enterprise class servers, some portion of the memory may not be licensed for usage by applications. Having unlicensed memory can reduce the initial cost of ownership and provides non-disruptive hardware upgrades over time.  The support of licensing of memory is referred to a Capacity On Demand (COD) or specifically for memory, Memory on Demand.  In situations where all the memory is licensed, the upper limit that can be assigned to applications is the actual amount of installed memory. In situations with unlicensed memory, the hypervisor is able to utilize memory from every installed DIMM (i.e. DIMMs are not treated as licensed or unlicensed), there is just an upper limit on the total amount of memory that can be assigned.

When a partition (virtual machine) is created, the hypervisor will chose processor and memory from chips/drawer that provide the best locality.  Usually the best possible performance is achieved when a partition is small enough to be contained in a single domain.  All of the memory reference therefore come from that single domain which improve the memory bandwidth and reduces the latency of requests from the processor for memory.  In the situation where there is dark memory, the probability that a partition can be contained to a single chip or a single drawer increases.  The more memory physically installed, the more degrees of freedom the hypervisor has in choosing the resources that can be assigned to the partitions on the server.

There are also reliability benefits from having unlicensed memory installed in the server.  With the advanced RAS available on Power servers, the hardware and software can in many cases predict which memory may fail in the future (e.g. based on soft but correctable errors).  If there is an indication that memory is starting to fail, the PowerVM hypervisor will copy the data from the memory locations that are failing to more stable memory locations.  Once all the data has been copied, the failing memory is garded (logically removed) so that it is not assigned to partitions.  The server is then protected from the failing memory which allows time for a repair action to be scheduled.

Unlicensed Processors
There are two different concepts of licensing processors.  One is referred to as software license restrictions where the operating systems and applications within the partition are limited to a fixed number of processors.  The other concept is hardware license restrictions where the server itself limits the number of cores that can be assigned to the partitions on the server.

Software Based Processors Limits
There are a couple ways to manage situations where there are software limits on processor usage.  One common way where there are software imposed limits is to use the Field Core Override (FCO) feature available from the Advanced System Management (ASM) interface.  Note, the FCO option is only available on the entry level Power servers; i.e. it is not available on the enterprise class servers.  From the HMC you can select Advanced System Management and after signing on to the server you can select the Field Core Override option.  The following is an example of the FCO panel (Note-You can only change the FCO setting when the server is powered off):

PowerVM FCOFor this server, there are 12 physical cores installed in the server but the FCO setting has been set to 4 so when the server is activated, only 4 of the 12 cores will be treated as installed. The remaining 8 cores will be logically removed (garded) and are not available for use.  The service processor will try to balance the number of activated processor cores across the active processor chips.  The FCO option is a very simple method of configuring your server to ensure compliance with software license limits.  One drawback of this approach is if the Power hardware detects that a processor may fail, the previously garded processors are not available as hot spares.  More information on FCO can be found at the following link.

Another approach to enforcing software processor limits is the use of shared processor pools.  A shared processor pool is a way to limit the amount of CPU consumption for a group of partitions.  From the HMC if you select Virtual Resources, Shared Processor Pool Management you are presented with:


From this panel you can set a maximum limit on processor consumption for a group of partitions.  You use the Partitions tab to indicate which partitions belong to specific shared processor pools.  The shared processor pool approach is a bit more complicated than the FCO option but it allows all the cores in the server to be available in the rare case of a processor failure.  Also, the maximum limit can be dynamically changed without requiring a server reboot.  More information on shared processors pools can be found at the following link.

Hardware Based Processor Limits (Capacity On Demand)
Hardware based processor limits, also referred to as COD, are limits on the number of physical processors available for applications.  When there are unlicensed cores, the Hypervisor will specifically choose individual cores to be licensed and unlicensed.  The hypervisor will put the unlicensed cores into a state that reduces the overall energy consumption of the server.  When a server is initially powered on or a customer runs a Dynamic Platform Optimizer (DPO) operation, the hypervisor treats all cores as licensed, makes the optimal assignment of processors and memory based on the configuration of the various partitions and then marks unused cores as unlicensed.  Having dark cores in the server allows the hypervisor to optimize the resource assignments yielding improved application performance.  Also, similar to dark memory, dark cores can be used as spares in the unlikely event of a processor failure.

 In summary, the PowerVM Capacity On Demand feature not only reduces the initial cost of ownership but allows customers to concurrently grow their hardware resources over time.  There are also performance and reliability benefits available on servers that have dark resources. More information about COD can be found at Power Systems Capacity On Demand.

Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions