IBM Security Verify

 View Only
Expand all | Collapse all

Memory Leak issue in ISVA 10.0.4.0 IF1

  • 1.  Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Fri April 07, 2023 11:39 AM

    Hello Team,

    We did an upgrade from 9.0.7.2 10.0.4 with IF 1. There is a clear indication memory spike issue on the reverse proxy instances. Created ticket, but not getting enough help from L2 & L3. L2 escalated to L3 but they are not convinced there is memory leak. 
    - Have done thorough analysis on reverse proxy session cache, worker thread and other tuning.
    - Memory keeps growing up even there is a 0 user load in system.

    I am adding the graph for last 30 days. On March 9th we did upgraded. I created a ticket after 2-3 days. but it's been close to 1 month. Am posting here if anyone can help in this community. Appreciate!

    Case No - TS012447418



    ------------------------------
    Bipin Dash
    ------------------------------


  • 2.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Mon April 10, 2023 09:14 AM

    I would take a support file when memory is high.  Unzip the support.txt from the support file, and check the support.txt and see what the process list shows as the top memory consumer.  Are you certain it is the reverse proxy instances (webseald) consuming the memory?

    Check to see if the webseald process is consuming memory or one of the java processes on the appliance (I am assuming virtual?).  Are there any other functions this appliance is performing?  Does it have the AAC or federation license installed?  If yes, then the runtime is likely running, so that could be the culprit.  Is the appliance part of a cluster?  Is it acting as a cluster master where it may be running the PD web runtime, DSC, config DB, and/or HVDB functions?  In any case, the support.txt is going to be your first point of reference to determine what process is consuming the memory.  You need to figure that out first unless you already know the webseald process is at fault.

    If webseald, then I would start looking at things like http transforms, user mapping rules, cert mapping rules, WCP/ModSec (likely not ModSec since that was a 10.0.5.0 feature unless you are using the technical preview) configuration.  ModSecurity was known to have a bad memory leak in v10.0.4.0 versions, but I am unsure if it is loaded without the advanced tuning parameter in v10.0.4.0.

    Also, have you considered just upgrading to v10.0.5.0 or is that not an option for you?  Good luck, I'm curious what you find out.  We're fighting a memory leak with the RTSS (AAC/Federation) right now that L2 hasn't been able to figure out yet.  Unfortunately the configurations are so unique in some environments it makes it pretty difficult to track things down sometimes.



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 3.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Mon April 10, 2023 11:20 AM

    Thank you Matt for you response.

    We don't have the AAC on the same WebSEAL virtual appliance. However we don't see any memory spike on AAC VA. 
    Yes, it's only for the Webseald process. It's cluster system with 2 Webseal VA  and we have DSC enabled. DSC process doesn't consume more memory. Below is the screen shot from the dynatrace monitoring system.


     The total memory consumption for webseald process is 3.03 GB, it keeps growing up over day by day (+200/300 MB). 
    We don't have any plan to go 10.0.5 for now because we have to rescope everything considering the time line. 

    I agree with you every environment is different. For time being, will wait for L2.  I will take a look regarding the modsecurity advanced tuning parameter if it gets enabled by default.

    Appreciate you response.



    ------------------------------
    Bipin Dash
    ------------------------------



  • 4.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Mon April 10, 2023 01:11 PM

    Bipin, very nice you have it tied into dynatrace.  Is that the dynatrace plugin that IBM has?  I saw that availability somewhere.  Very useful in this case!

    That's interesting webseald would keep growing.  What are your session timeouts?  I am just thinking maybe something where user authentication load leaving sessions open could be consuming memory.  Does it seem to grow in relationship with request and/or authentication requests where you could corelate it to perhaps the mechanism inside webseal that could be causing the issue?



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 5.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Mon April 10, 2023 03:40 PM

    Matt, Dynatrace provides this licensed agent to deploy in ISVA. You can deploy this agent in System -> extension  and also need to specify API key and credential details.

    Session time out is 1800 Secs. Even in week end, the user count is 0 and DSC session count is also 0, still there are spikes. During business day, the user load is pretty less, max 100 users. I did collect some pdweb.debug, threads and snoops. and didn't find any corelation, was thinking could be hgher usage of worker threads but doesn't look like that's the reason.



    ------------------------------
    Bipin Dash
    ------------------------------



  • 6.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Tue April 11, 2023 08:49 AM

    The only other thing I can think is maximum SSL/TLS sessions, [ssl] ssl-max-entries being extremely high, and then having a load balancer doing full SSL/TLS handshakes in front where it does not keep track of the SSL session ID.  However, I would think you would have to be aggressively doing health checks and have those max-entries set very high at the same time to use the amount of memory you are seeing consumed.  Your increase looked pretty steady except in a couple places, likely influx of user traffic.  But a steady increase would make me think of something that was constantly running against the webseal.

    Also, from what you were describing about the dynatrace extension I think that is the same one I was thinking of.  Very useful info!

    I just had a thought, any chance the dynatrace monitor could be pulling stats and causing a memory usage when those stats are pulled?  That is one thing I was just thinking you have different that not a lot of others likely use where this would not have been seen before.



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 7.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Mon April 17, 2023 06:10 PM
    Edited by Bipin Dash Mon April 17, 2023 06:10 PM

    I did some tuning related to Worker threads, max-ssl entries, it helps to slow down % of Memory spike but doesn't mitigate the issue.  I don't think dynatrace i.e. oneagent takes very less memory. if you see my previous screen shot, dyntrace process is oneagent, it's in some megabytes



    ------------------------------
    Bipin Dash
    ------------------------------



  • 8.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue April 11, 2023 08:51 AM

    Hi,

    I'm  experiencing same issue on our 5 environments (TS011794922) - memory utilization of webseald is growing from moment when WebSEAL is started till it crash. This recommendation from L3 may be helpful:

    ". . . the recommendation is to set webseal environment variable MALLOC_CHECK_=3 in as many instances as you are comfortable setting it. This does some sanity checking of the memory operations. If webseal tries to do something "bad" with the memory, it can cause webseal to crash and generate a stack dump which can then give us information to help debug that issue.

    Along with that, to debug the memory usage we need to understand how the memory is used or allocated. . . . "



    ------------------------------
    Piotr Dąbrowski
    ------------------------------



  • 9.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue April 11, 2023 09:28 AM

    Thank you Piotr!, I will post this information to PMR if it's helpful.



    ------------------------------
    Bipin Dash
    ------------------------------



  • 10.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue April 11, 2023 08:51 AM

    Hi,

    I'm experiencing same issue on our 5 environments - memory utilization of webseald grows from moment that WebSEAL is started till it crash. Issue had been present on both 10.0.4 and 10.0.5. (Support case where we reported it is TS011794922).

    I'm sharing recommendations from L3:

    " . . . the recommendation is to set webseal environment variable MALLOC_CHECK_=3. in as many instances as you are comfortable setting it.    This does some sanity checking of the memory operations.   If webseal tries to do something "bad" with the memory, it can cause webseal to crash and generate a stack dump which can then give us information to help debug that issue.

    Along with that, to debug the memory usage we need to understand how the memory is used or allocated. . . ."


    ------------------------------
    Piotr Dąbrowski
    ------------------------------



  • 11.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue April 11, 2023 08:50 AM

    We are also facing memory leak issue with ISVA 10.0.5 on AAC/Federation (there was no issue with 10.0.4). Which version do you have this memory leak with?



    ------------------------------
    Re Ad
    ------------------------------



  • 12.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue April 11, 2023 09:30 AM

    This is specific to WebSEALs on 10.0.4.0 IF1. 



    ------------------------------
    Bipin Dash
    ------------------------------



  • 13.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Tue April 11, 2023 09:36 AM

    AAC/Federation definitely has a memory leak between v10.0.4.0 and v10.0.5.0.  If you take a support file, you can see the java process for the runtime consuming a large amount of memory.  The process will eventually consume all available system memory and crash the process.  It will restart, but we witnessed some oddities with the virtual appliance once this crash occurred.

    We have a support case open with Annelise with L2 on this.  Annelise believes it could be related to tracing.  I need to update that case today as a colleague of mine was reviewing our trace spec and we had a good amount of trace turned on in some of our environments.  From what Annelise had discovered in her lab, she had initially suspected the tracing.

    Out of curiosity, do you have tracing enabled and do you have a lot of things tracing on your runtime?



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 14.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Fri May 05, 2023 09:16 AM

    Yes, tracing is enabled on runtime.
    We installed ISVA 10.0.5 Fixpack 1 but it did not solve the issue, we are still observing leaking memory.
    Did you receive any update on your open case?



    ------------------------------
    Re Ad
    ------------------------------



  • 15.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Fri May 05, 2023 10:00 AM

    L2 is still looking into it.  They unfortunately have not been able to determine the cause yet.  We have also seen the same issue on our environments that utilize containers.



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 16.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Wed May 24, 2023 12:41 PM

    @Matt Jenkins I'm facing on same issue. Contact me for more details.

    Regards



    ------------------------------
    Valentino Carnovale CSM - Architect Security
    ------------------------------



  • 17.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Fri June 16, 2023 10:39 AM

    No update in PMR as of now. The CPU and Memory goes crazy (>90%) and when we do a load testing of single sign on (http header based) of 100K user in 1 hour. Page doesn't load. AVG response time goes from 5 secs to 20 secs. The CPU comes down when we stop the load testing but memory doesn't go down. It's large memory leak resides in ISVA 10.X version. Unfortunately IBM/L2 team is unable to find any fix and it's impacting customers.



    ------------------------------
    Bipin Dash
    ------------------------------



  • 18.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Mon June 26, 2023 07:41 AM

    Hello isam'ers,

    We have recently upgraded our production environment to v10.0.5 IF1 (6 weeks ago) and we are also observing memory leaks.

    ISVA seems to never free any memory whatsoever, leading to a situation where a process is killed to free some memory, either the DSCD or the Java processes, which has a direct impact on our customers.

    To avoid that, we are monitoring memory usage daily and we have to perform a manual weekly reboot.

    By reading your comments, I'm not sure opening another new PMR about the same issue will help but I will be doing it anyway.

    In the meantime, if any of you has new information related to this issue, it would be greatly helpful for us.

    Regards



    ------------------------------
    André Leruitte
    ------------------------------



  • 19.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Mon June 26, 2023 08:50 AM

    In v10.0.6.0 there was an APAR for DSCD:

    IJ46144 DSC MEMORY CONSUMPTION ISSUE

    For RTSS, there is a fixpack L2 has that we have been testing.  So far it has seemed to plug the larger leak L2/L3 refers to.

    "L3 has generated a Fixpack for 10.0.5 that will upgrade Semeru J9 JRE to version 11.0.19+7 Open J9 0.38.0."

    Annelise from L2 is aware of the fixpack.  They may have a fixpack for DSCD on v10.0.5.0 as well, IDK.  But right now the suggested route would likely be update to v10.0.6.0.  However, the v10.0.6.0 update doesn't include the fix for the major RTSS leak.



    ------------------------------
    Matt Jenkins
    ------------------------------



  • 20.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    Posted Tue June 27, 2023 11:28 AM

    We last upgraded from 9.0.7 to 10 on our way to 10.0.2. We have had the product since it was TAMeb and had gone to virtual appliances, upgrading from 9.0.2 > 9.0.3 > 9.0.7 > 10 > 10.0.2.  One interesting thing we encountered while working a PMR with IBM was the defaults had been changed multiple times since our original configurations were created. The suggestion was to create a new instance to generate a fresh configuration, then diff the new defaults with our existing ones. I don't think that will cure a memory leak, but it will significantly bump up several defaults that were set many years ago when memory was low and expensive. 

    In  the meantime, my colleagues and I are watching this thread as we recently upgraded our development appliance to 10.0.5 and are holding off on doing our staging and production appliances if there are complaints of memory leaks.  I like the  tip to take a support file and we weren't aware there was a plugin available and supported from Dynatrace. We currently  use the product for our enterprise java apps. Good to know. As usual, the community is a great source for ideas and tips. 



    ------------------------------
    David Gianetti
    ------------------------------



  • 21.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Thu June 29, 2023 03:09 AM

    Hi all,

    It seems IBM has identified a memory leak in webseal affecting v10.0.4, 10.0.5 and 10.0.6 : IJ47321: REVERSE PROXY MEMORY LEAK PROCESSING JSON ARRAY DATA (ibm.com)

    We think there also are memory leaks in the acc/federation runtime that still have to be found, but at last it is a step in the right direction. Now we will still have to wait for v10.0.7 to finally have a fix.

    In the meantime, we are going to implement an nightly automatic restart of all our reverse proxies and of the runtimes.



    ------------------------------
    André Leruitte
    ------------------------------



  • 22.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Thu June 29, 2023 08:00 AM

    Hi again,

    2 new APAR's spotted today related to memory leaks : 

    • https://www.ibm.com/support/pages/apar/IJ47335?myns=swgimgmt&mynp=OCSSRGTL&mync=E&cm_sp=swgimgmt-_-OCSSRGTL-_-E
    • https://www.ibm.com/support/pages/apar/IJ47347?cm_sp=swgimgmt-_-OCSSRGTL-_-E&mync=E&mynp=OCSSRGTL&myns=swgimgmt

    Regards



    ------------------------------
    André Leruitte
    ------------------------------



  • 23.  RE: Memory Leak issue in ISVA 10.0.4.0 IF1

    IBM Champion
    Posted Fri July 07, 2023 06:47 AM

    Follow-up on the deployment of v10.0.6, which fixes one of the memory leaks: we started deploying in our DEV and TEST environments, but we are now blocked as v10.0.6 breaks any FIDO2 authentication : IJ47417: FIDO2 AUTHENTICATION FAILURES IN ISVA 10.0.6.0 (ibm.com)

    We are stuck on v10.0.5, being forced to daily restart webseal and federation runtime.

    If anyone has any better solution/workaround, I am interested :)  



    ------------------------------
    André Leruitte
    ------------------------------