Turbonomic

 View Only
  • 1.  How to mitigate the out-of-memory issue

    Posted Thu June 22, 2023 10:11 AM

    Hi,  we opened a ticket to report an issue where some API Connect failed to get analytics, the Support found out it's due to the large payloads which caused out of memory, and guided us to increase the pod's heap memory limits. So my question:
    How Turbo is able to prevent the out-of-memory issue and reduce the support ticket? Will it detect in time the out of memory risk and does it allow to automatically adjust the memory limits when necessary to avoid the app hanging like this? Thank you.



    ------------------------------
    Peining Tsou
    ------------------------------


  • 2.  RE: How to mitigate the out-of-memory issue

    Posted Fri June 30, 2023 08:26 AM

    Hello Peining, 
    Starting Turbonomic version 8.9.2's OVA, it will include a new container monitoring notification for memory/restarts.  This only applies to new OVA deployments.  



    ------------------------------
    David Zhou
    ------------------------------



  • 3.  RE: How to mitigate the out-of-memory issue

    Posted Mon July 03, 2023 02:45 PM

    Thanks David. 



    ------------------------------
    Peining Tsou
    ------------------------------



  • 4.  RE: How to mitigate the out-of-memory issue

    Posted Fri June 30, 2023 03:25 PM

    Hi @Peining Tsou what type of containerized environment do you have API Connect deployed in?  If you have Turbo deployed we can also target the Kubernetes/OpenShift cluster with KubeTurbo detailed here: https://www.ibm.com/docs/en/tarm/latest?topic=configuration-cloud-native-targets that can size your pods to help prevent out-of-memory issues.  Glad to discuss further



    ------------------------------
    Jason Shaw
    Delivery Consultant | IT Automation
    IBM Technology Expert Labs
    ------------------------------



  • 5.  RE: How to mitigate the out-of-memory issue

    Posted Mon July 03, 2023 03:18 PM

    Many thanks Jason. It's on OCP.  Here is more background info:

    We increased pod memory limits to 2G and LS JAVA options to -Xms1g -Xmx1g

    Even then pod continued to restart after 10-15 min.

    We have recently pushed some APIs which have large payloads. The Support asked to turn off payload logging in the Activity-log for success cases for those APIs. It worked for a few days, but Analytics service going down very frequently.. Also, after rebooting Analytics service, it's not coming up, The connection to the server localhost:8080 was refused. Afterwards, we found out the ISO was not mounted in vmware. Wondering if Turbo with KubeTurbo can help prevent all of these. Additionally, my understanding is to set limits for memory/heap. What if we push large payloads suddendly that are over the limits (I heard Turbo will respect the limits), can Turbo still resize smoothly without OOO?



    ------------------------------
    Peining Tsou
    ------------------------------



  • 6.  RE: How to mitigate the out-of-memory issue

    Posted Wed July 05, 2023 11:46 AM

    In short yes, Turbonomic with KubeTurbo deployed in the OCP cluster will give your actions/recommendations to automate the sizing of your workload based on utilization in effort to prevent OoM.  For more details see the documentation here on the Monitored Resources and Actions: https://www.ibm.com/docs/en/tarm/latest?topic=configuration-cloud-native-targets 



    ------------------------------
    Jason Shaw
    Product Manager | Turbonomic
    IBM Software | IT Automation
    ------------------------------