PowerVM

 View Only
Expand all | Collapse all

PowerVM Hypervisor memory

  • 1.  PowerVM Hypervisor memory

    Posted Tue March 07, 2023 09:32 AM

    Is it possible to dynamically reduce the reserved memory for Hypervisor? I have a system that was using VIOs that has since been rebuilt as a standalone full system resource partition. Reserved memory is showing as several Gb and could be needed in this environment due to workload, lack of live partition mobility and business outages (so partition can not be moved with an outage).

    The previous VIO partitions are still there but not activated. Could removing these reduce reserved hypervisor memory?#

    Thanks



    ------------------------------
    Dave Heald
    ------------------------------


  • 2.  RE: PowerVM Hypervisor memory

    Posted Wed March 08, 2023 01:40 AM

    Hi!

    Reserved memory for hypervisor is not directly influenced by the presence of VIO servers, as they use normal RAM.

    In my experience things that contribute to hypervisor RAM usage is the RAM and CPU scalability of each partition.

    Lets say you have a partition using 10 vCPU (Desired) and Max is 30, then hypervisor RAM has to be allocated for the dynamic scalability ot 30 vCPU.

    This is especially noticable for RAM. So if your application requires 100GB RAM (Desired) but you have given it 200 GB Max, apx 4% of that 100GB is reserved for the page table entries by the hyervisor.

    Hypevisor Mirroring if enabled  also contributes to RAM usage.

    SR_IOV Network adapters reserve RAM.

    If this is an older server, Disable Hypervisor IO adapter enlarged capacity, as descibed on page 59 here:

    https://ibm.ent.box.com/v/powerbestpractice

    So the most effective way to cull hypervisor RAM usage is to keep MAX and DESIRED values of RM as close as possible, without sacrificing application availability.



    ------------------------------
    FREDRIK LUNDHOLM
    ------------------------------

    Original Message


  • 3.  RE: PowerVM Hypervisor memory

    Posted Wed March 08, 2023 09:22 AM

    goodday,

    Some more info on the PageTable calculation:

    In the ASMI interface you can set the LMB size (Logical Memory Block). If this is set at 256MB you can increase lpar memory in steps of 256MB. This LMB size is also being used for the calculation of the AIX PageTable (each AIX lpar has a PageTable). This size is calculated by dividing the max memory setting in your profile , divided by the LMB size. So if you have max 64GB , then 64GB / 256MB = 256MB AIX Page Table which is rather large and most probably overkill. This AIX PageTable is take from real memory and in my believe is part of the Hypervisor memory. So if you have 10 lpar with each 256MB pagetable , then hyporvisor memory is 10 x 256MB.

    Your other remarks makes also perfectly sense!

    Regards

    Michel.



    ------------------------------
    Michel de Kraker
    ------------------------------

    Original Message


  • 4.  RE: PowerVM Hypervisor memory

    Posted Thu March 09, 2023 09:23 AM

    Many years ago I was told that the size of a hypervisor page table for an LPAR is always rounded up to a power of two.  Example: an LPAR with a maximum memory of 33GB has the same size HPT as an LPAR with a maximum memory of 64GB.  The moral: You should always set the maximum memory to an even power of two.

    Is this correct?  Was it ever correct?



    ------------------------------
    Anker Lerret
    Blue Cross Blue Shield of Massachusetts
    ------------------------------

    Original Message


  • 5.  RE: PowerVM Hypervisor memory

    Posted Thu March 09, 2023 09:34 AM
    I remember exactly the same! 

    The reason I raised this question is that I wasn't here for whatever previous decisions and migrations affecting this server were made. I was asked the question on my second day on site if the LPAR could be increased by 40Gb - can find 39Gb by removing the redundant VIO servers plus free mem on the frame,so wondered if the hypervisor memory was now configurable!

    Notice:
    This email is confidential and may contain copyright material of members of the Ocado Group. Opinions and views expressed in this message may not necessarily reflect the opinions and views of the members of the Ocado Group.

    If you are not the intended recipient, please notify us immediately and delete all copies of this message. Please note that it is your responsibility to scan this message for viruses.

    References to the "Ocado Group" are to Ocado Group plc (registered in England and Wales with number 7098618) and its subsidiary undertakings (as that expression is defined in the Companies Act 2006) from time to time. The registered office of Ocado Group plc is Buildings One & Two, Trident Place, Mosquito Way, Hatfield, Hertfordshire, AL10 9UL.




    Original Message


  • 6.  RE: PowerVM Hypervisor memory

    Posted Wed March 08, 2023 03:56 AM

    Hi,

    as Frederik stated, it is not possible to modify the Hypervisor Memory usage. Hypervisor "owns" that and needs it and it could also happen, that a newer System Firmware that will be installed might own more Hypervisor Memory due to enhancements done. So the available Memory planning must be done in advance.
    There is a Technote about Hypervisor Memory usage:
    https://community.ibm.com/community/user/power/blogs/pete-heyrman1/2020/06/08/understanding-powervm-hypervisor-memory-usage

    The System planning tool could be used to get a possible calculated amount of Hypervisor Memory reserved to know, how much memory you actually have left for your LPARs.

    When the System was never planned for VIOs / more Client LPARs then you need to live with the current setup or , when it is not a too old system, maybe buy some bigger Memory add on.

    Greetings

    Marc-Eric Kahle



    ------------------------------
    Marc-Eric Kahle
    ------------------------------

    Original Message


  • 7.  RE: PowerVM Hypervisor memory

    IBM Champion
    Posted Wed March 08, 2023 04:26 AM

    Hi Dave,

    even if I don't know a way how to dynamically decrease memory reserved by hypervisor, but there is a small trick. If your VIOs were overdimensioned (max RAM was too high), you can change the memory values in their profiles - let's say 256MB if you don't need them, and then start them from new profiles in OpenFirmware mode. In this moment hypervisor recalculates the reserved memory and you should see it lower. Then you can shutdown your VIOs again if you don't need them. I didn't test on the newest systems (Power10), but it did help with older systems.



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------

    Original Message


  • 8.  RE: PowerVM Hypervisor memory

    Posted Wed March 08, 2023 04:38 AM
    Thanks all for your replies.

    Will certainly try the VIO profile change! Not sure when I will be able to try it but will update when I can.

    Thanks again

    Notice:
    This email is confidential and may contain copyright material of members of the Ocado Group. Opinions and views expressed in this message may not necessarily reflect the opinions and views of the members of the Ocado Group.

    If you are not the intended recipient, please notify us immediately and delete all copies of this message. Please note that it is your responsibility to scan this message for viruses.

    References to the "Ocado Group" are to Ocado Group plc (registered in England and Wales with number 7098618) and its subsidiary undertakings (as that expression is defined in the Companies Act 2006) from time to time. The registered office of Ocado Group plc is Buildings One & Two, Trident Place, Mosquito Way, Hatfield, Hertfordshire, AL10 9UL.




    Original Message


  • 9.  RE: PowerVM Hypervisor memory

    IBM Champion
    Posted Wed March 08, 2023 07:14 AM
    Edited by Satid Singkorapoom Wed March 08, 2023 08:09 AM

    Dear Dave

    In 2014 (POWER7+ time),there was an article in IBM Systems Magazine providing explanation on Hypervisor's memory usage and a guideline for setting Max Memory parameter of each LPAR profile so that Hypervisor does not unnecessarily consume extraneous memory of the server.  This is in line with other responses you received and I hope this guideline is still valid for recent Power Systems models.  

    Guidelines for MaxMem


    Based on the information provided in this article, you'll want to follow these guidelines:

    • Don't set MaxMem equal to total amount of server memory for each LPAR. This will waste memory with oversized HPTs.

    • Size MaxMem approximately 25 percent larger than Desired Memory for production LPARs that might require a DLPAR memory addition. For less important LPARs, which could be shutdown for a desired memory increase, set the MaxMem as close as possible to the Desired Memory.

    • Always set the MaxMem to a power of two to maximize DLPAR growth options.    



    >>>>  The previous VIO partitions are still there but not activated. Could removing these reduce reserved hypervisor memory? <<<<

    Comparing to reducing the Max Memory parameter of all LPAR profiles, deleting all unused LPAR profiles should give you back the most amount of extraneously allocated server memory. 

    ------------------------------
    Education is not the learning of facts but the training of the mind to think. -- Albert Einstein.
    ------------------------------
    Satid S.
    ------------------------------

    Original Message


  • 10.  RE: PowerVM Hypervisor memory

    Posted Thu March 09, 2023 09:17 AM
    Edited by MARC SALAVA Thu March 09, 2023 09:33 AM


  • 11.  RE: PowerVM Hypervisor memory

    Posted Thu March 09, 2023 09:32 AM

    As someone who has worked both as a customer and and IBMer, here are my recommendations for setting Max memory

    You need to find a proper balance between the need to potentially increase the allocated memory and the pain involved in getting an outage to change the Max setting.

    If outages are easy to come by, then I would set the Max at or very close to the Desired (allocated) memory. If there is a need to increase the allocation, you can change the Max at your next outage.

    If outages are difficult to come by (I worked in an environment where, for certain LPARs, one outage per year was the norm), then you need to have a larger delta between the Desired and Max to accommodate any requests for more memory until the next outage.

    In the end, most LPARs are somewhere in the middle, so you want to have some delta between the Desired and Max. How large of a delta depends on how likely there will be a demand for more, compared to the ability to take an outage. In no case would I set the Max higher than necessary for increases during that interim. Not only does this increase Hypervisor memory usage, it also causes a performance hit as an unnecessarily large HPT takes more time to scan.

    I use "rules of thumb" when I don't have actual data to use when making my decisions and then dial it in more accurately as that data is gathered. In no case would I ever set the Max to a value larger than I could potentially need before the next available outage window.



    ------------------------------
    MARC SALAVA
    ------------------------------

    Original Message


  • 12.  RE: PowerVM Hypervisor memory

    Posted Thu March 09, 2023 02:46 AM

    Hi Andrey,

    I wonder if, in this case, this is true. The way I read the question is that the definitions of the viosses stil exist. They are not running. That is not possible I guess because the running partion has all the resources. (I assume this partion has the 'Assign All System Resources' box checked in its profile)

    My gut feeling is that removing the vios definitions won't free any memory.



    ------------------------------
    Ane-Pieter Wieringa
    ------------------------------

    Original Message


  • 13.  RE: PowerVM Hypervisor memory

    IBM Champion
    Posted Thu March 09, 2023 04:33 AM
    Edited by Andrey Klyachkin Thu March 09, 2023 04:37 AM

    Hi Ane-Pieter,

    if the box "Assign All System Resources" is checked, then you are right. The VIO LPARs won't start because they will not find any resources.

    But if they can start, reducing max memory for the LPARs will reduce the size of memory consumed by hypervisor.

    I did a small test now with HMC V10R1SP1020 and IBM Power E1080 9080-HEX FW1010.31.

    1. Get resource dump of memory consumption and check it, how many LMBs the hypervisor uses:
    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      184        212          3

    Target LMBs are 212. It is the same memory as it shown in HMC as Reserved Memory, multiplied by LMB size. The LMB size is 256MB in this case.

    2. I have a switched off LPAR (without any OS) which is configured to 1GB desired memory and 4GB maximum memory. I changed the maximum memory to 8GB and switched on LPAR into OpenFirmware mode. Then I took another dump:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      184        214          3

    As you can see the number of Target LMBs is increased by 2 LMBs.

    3. Then again I switched off the LPAR, changed maximum memory setting back to 4GB, switched on the LPAR into OF mode and then switched off again. Another dump shows that Target LMBs came back to 212:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      184        212          3

    Another test. The same system but another time:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      186        214          3

    I create a new empty LPAR without any I/O resources, just 0.1 CPU, 1GB RAM desired and 4GB RAM maximum. The LPAR is not started:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      186        216          3

    Hypervisor memory consumption is increased by 2 LMBs. I started the LPAR:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      186        216          3

    Memory consumption is the same as it was. Hypervisor reserved the memory during the creation and didn't change it after starting LPAR. Now it is time to shut down the LPAR:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      186        216          3

    LPAR is down but hypervisor still reserves the memory it needs if the LPAR will be started. I delete the LPAR and the memory is freed:

    LpId    HvLpEntry@      Group  InitLmbs  MinLmbs  MaxLmbs  CurLmbs TargetLmbs UsableLmbs
   0 8004000013E81000   *None        1        1    65536      186        214          3

    Of course it is not "the official IBM method to reduce the hypervisor memory". I am not an IBM employee and it is just experience and what I learned from more experienced colleagues. It may work in your environment but it is not guaranteed. As Frederik and others already wrote, the hypervisor memory consumption may be result of many factors. Not all of them can be changed online or changed at all.



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------

    Original Message