WebSphere Application Server & Liberty

WebSphere Application Server & Liberty

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Back to discussions
Expand all | Collapse all

Memory over-commitment with Liberty Containers on Kubernetes plattforms

  • 1.  Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/18/26 01:06 PM

    We are deploying multiple Liberty Pods into one namespace on OpenShift. The containers are currently running the follwing Liberty and JDK versions: 

    product = Open Liberty 26.0.0.1 (wlp-1.0.109.cl260120260112-1902)
wlp.install.dir = /opt/ol/wlp/
server.output.dir = /opt/ol/wlp/output/defaultServer/
java.home = /opt/java/openjdk
java.version = 17.0.17
java.runtime = IBM Semeru Runtime Open Edition (17.0.17+10)

    Currently we are following the Liberty best practices and configure the pods with requested memory equals memory limit. I.e.

          resources:
        limits:
          memory: 2Gi
        requests:
          cpu: 250m
          memory: 2Gi

    Furthermore we are using the JVM options -XX:MaxRAMPercentage=75 so that the JVM starts with a max heap sized 75% of the memory limits i.e. the MaxHeap used is: `1,610,612,736` Bytes. 

    Due to the high number of pods and the setting `resources.requests.memory = resources.limits.memory` we are allocating lots of memory which is often not needed as the pods most of the time use just ~1GB memory. 

    We are fully aware of the difference of `resources.requests.memory` and `resources.limits.memory` and that there is no guarantee to get `resources.limits.memory` allocated. But a current analysis of the cluster shows there there are no memory bottle-neck to be expected in the cluster. So we have a good chance that we would have `resources.limits.memory` available if needed. 

    So we are considering changing the resource settings to: 

          resources:
        limits:
          memory: 2Gi
        requests:
          cpu: 250m
          memory: 1Gi


    Using this configuration the pod might start with 1GB memory however max heap being set to 1,610,612,736 bytes (). 

    So I'm wondering  what happens if the applications over time needs more java heap memory as can fit in the guaranteed 1GB (/sys/fs/cgroup/memory.current shows at the startup 414662656) pod memory - assuming the pod can get up to 2GB of memory from Kubernetes? Can we expect that the pod requests more memory from Kubernetes (namely up to 2GB) and enhance java heap up to 1,610,612,736 bytes?



    ------------------------------
    Hermann Huebler
    Cloud Architect
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------


  • 2.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/19/26 06:40 AM

    What normally happens in K8S is that if there is not enough memory during the run of a Pod to satisfy the memory requirements, then an OOMKilled is issued and the Pod dies and will try to restart.  The Pod itself can go up to 2GB of memory, but if it cannot get that memory it will be killed.  As you rightfully state, the memory for the heap will never grow more than 1.610.612.736 bytes.

    Brian



    ------------------------------
    Brian S Paskin
    Sr. Technology Engineer
    IBM Cloud Engineering
    ------------------------------



  • 3.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/19/26 01:32 PM
    Edited by Daniel Pfeifer 03/20/26 06:32 AM

    Since memory is not really compressible you may run in one of two issues:

    - If the node doesn't have whatever the JVM requests to grow, GC will trigger. If that won't help, you will get an OutOfMemoryError.
    - If the node is under pressure, it may kill the most "misbehaving" pods first, i.e. it prioritzes eviction of pods that are too far off from what they actually requested.



    ------------------------------
    Daniel Pfeifer
    Principal Consultant
    Whistler AB
    ------------------------------





  • 4.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 05:57 AM

    Hello  @Daniel Pfeifer and @Brian S Paskin,

    many thanks for your responses on this one. Really appreciate 

    So to summarize what can we expect from Semeru in case the pod starts with the requested resources is:

    • If we reach the  current heap size we can expect that the GC kicks in and if required tries to expand the Heap (up to the max heap)
    • If the expanded heap requires more memory the pod requests more memory from K8S up to the memory limit 
    • If K8S can't provide memory up to the pods memory limit the pod dies with OOMKill 
    • If the pods gets memory up to it's memory limit but the Heap request can't be satisfied we get an OOM (as standard JVM behavior) 
    • If the pod reached it's memory limit and requests more (for example for native memory) we get an OOMKill (as standard K8S behavior) 
    • If the node itself is under memory pressure those pods which are "overshooting" the requested memory by most get evicted 

    Is that some the correct understanding? 

    Thanks again and have a good day, Hermann



    ------------------------------
    Hermann Huebler
    Cloud Architect
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------



  • 5.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 06:26 AM

    Looks correct to me.  If the Node itself is under pressure there are certain strategies that can be deployed.  However, with observability tools, placement of the Pods can be optimized.  See Turbonomics.



    ------------------------------
    Brian S Paskin
    GTM Product Manager, Application Runtimes
    IBM
    Roma, Italy
    ------------------------------

    Original Message


  • 6.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 06:32 AM
    Edited by Daniel Pfeifer 03/20/26 06:34 AM

    Hi Hermann,

    there were some issues yesterday with me posting. I tried three times and only the one (with much less information) eventually went through, I don't know which black hole swallowed the more detailed one, I will post it once more in the end of this message just for reference, but let me address your bullet points first:

    1. Since you set MaxRAMPercentage (without setting Min/InitialRAMPercentage) the JVM's heap will eventually grow to whatever you set CGroups Limit you set (-25% of course). To be fair, I haven't checked if it will try to GC before expanding the heap on Semeru GenCon, but it will definitely try to start GC:ing if heap is expanded to the maximum and it needs to clear up space. I think in the end it doesn't matter what comes first in this case.
    2. Yes
    3. Yes
    4. To an extent. It won't grow more than MaxRAMPercentage (it won't even try), at that point it will try to start spending a fair amount of time on GC and if it can't reclaim enough resources for your application to work, it will fail with an OutOfMemoryError. So in this case it won't be killed off by K8S, the JVM will just give up.
    5. Yes
    6. Yes

      

    Now just for completeness my original message which had a bit more details:

    Memory is not a compressible resource, so one of two things may happen:

    • If the node has no more RAM to give and your heap tries to grow then your JVM may try to GC first, but will eventually crash with an OutOfMemoryError when RAM that it believes to be available can not be used.
    • If a node is under pressure, it will evict nodes based on who it deems to be most ill-behaving, prioritizing those that are exceeding their requested resources the most, so you can only hope someone else is more ill-behaving than your pods before the scheduler reaches your pods.

    There are minutiae in all this, you could probably play with setting -XX:InitialRAMPercentage=75 -XX:MinRAMPercentage=75, just to make sure that the pods can straight up allocated enough RAM when they start, but it would still not stop the node from evicting your pods if the scheduler deems your pods to be the most ill-behaving… and of course, you wouldn't really conserve RAM either, it has allocated 75% of the CGroups Limit at that point.

    Grüsse in die Kaiserstadt!

    /D



    ------------------------------
    Daniel Pfeifer
    Principal Consultant
    Whistler AB
    ------------------------------

    Original Message


  • 7.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 08:45 AM

    Hello @Daniel Pfeifer and @Brian S Paskin,

    thanks again for your response here . Appreciate your review and further comments.

    @Daniel Pfeifer - regarding the max heap size - although I set the -xx:MaxRAMPercentage=xx only this one is translated  xx% of the pods memory limit. Verified this via the verboseGC log after the restart of the pod. So this means to me that at startup of the pod the JVM looks for the memory limit of the pod ( by default UseContainerSupport is enabled) and then internally sets the -Xmx to the calculated value. 

    Danke für die netten Grüße in den hohen Norden :) 

    @Brian S Paskin - thank for mentioning Turbonomics.  Sounds interesting - but we don't have it available so far.


    Thanks again and have a nice weekend, Hermann



    ------------------------------
    Hermann Huebler
    Cloud Architect
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------

    Original Message


  • 8.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 11:01 AM
    Edited by Gary DeVal 03/20/26 11:10 AM

    A point of clarification: Semeru checks the available system memory only once, when the JVM starts up. It does not monitor available system memory as the JVM is running. 

    So if the pod increases (or decreases) memory at some point after the JVM starts up, Semeru will not know that and will not adjust Xmx to reflect the new available memory in the pod.

    Regards,
    Gary DeVal

    WebSphere/Java Performance Engineering



    ------------------------------
    Gary DeVal
    ------------------------------

    Original Message


  • 9.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 11:55 AM

    Thanks a lot for pointing this out  @Gary DeVal. But from my testing it seems that Semeru does not check the (currently avalable) System Memory to determine the max heap but the pods memory limit and internally sets the -Xmx to this calculated value. Right?

    Have a good weekend, Hermann



    ------------------------------
    Hermann Huebler
    Cloud Architect
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------

    Original Message


  • 10.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 12:20 PM

    The JVM will look at the cgroup memory limit (if any) to determine the available memory for the purpose of calculating -Xmx. 

    I am not so familiar with how Kube translates resource limits like memory request and memory limit into cgroup settings, so I asked Copilot about it.  :)

    #####
    Q: If a kubernetes pod is started with memory request 1GB and memory limit 2GB, what does the cgroup information for the pod show as the memory limit?

    A: In Kubernetes, the memory request has no effect on the cgroup memory limit.
    Only the memory limit is enforced by the container runtime and shown in the pod's cgroup configuration.

    If you start a pod with:

    memory request: 1Gi

    memory limit: 2Gi

    Then the pod's cgroup will show the memory limit as 2Gi, not 1Gi.

    #####

    This aligns with your testing observation. The pod memory request is invisible to the JVM, so it calculates Xmx based on the memory limit.

    Cheers,

    Gary DeVal

    WebSphere/Java Performance Engineering



    ------------------------------
    Gary DeVal
    ------------------------------

    Original Message


  • 11.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 12:31 PM
    Edited by Daniel Pfeifer 03/20/26 12:31 PM

    I think you guys are just agreeing to disagreeing. CGroups Limits is the definitive upper limit. This does not change unless you change the config of the pod (which triggers a restard either way).



    ------------------------------
    Daniel Pfeifer
    Principal Consultant
    Whistler AB
    ------------------------------

    Original Message


  • 12.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 01:02 PM

    Well - I see it more positively - the pod's memory limit gets translated to the cgroup memory limit by K8S and Semeru uses the cgroup's memory limit to calculate the -Xmx if -XX:MaxRAMPercentage=xx is set at pod startup.

    I think we are all on the same page now :)



    ------------------------------
    Hermann Huebler
    Cloud Architect
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------

    Original Message


  • 13.  RE: Memory over-commitment with Liberty Containers on Kubernetes plattforms

    Posted 03/20/26 05:56 PM

    Yes, as far as I'm concerned we are agreeing. 😁

    My original "clarification" was prompted by my misunderstanding of how Kube handles differing memory request and limit values; I thought the pod would start with only the request memory "available", and would have to increase the "available memory" for the pod to grow to the limit value. TIL that the pod will be launched with the full memory "available" (based on the cgroup limit) so the value used by the JVM to set Xmx will be the pod limit value, not the request value. Therefore, although my comment about how Semeru sets Xmx is correct, that is not relevant to Hermann's question.

    Sorry if my input caused any confusion. 

    Cheers.



    ------------------------------
    Gary DeVal
    ------------------------------

    Original Message


Related Content