I did some tuning related to Worker threads, max-ssl entries, it helps to slow down % of Memory spike but doesn't mitigate the issue. I don't think dynatrace i.e. oneagent takes very less memory. if you see my previous screen shot, dyntrace process is oneagent, it's in some megabytes
Original Message:
Sent: Tue April 11, 2023 08:49 AM
From: Matt Jenkins
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
The only other thing I can think is maximum SSL/TLS sessions, [ssl] ssl-max-entries being extremely high, and then having a load balancer doing full SSL/TLS handshakes in front where it does not keep track of the SSL session ID. However, I would think you would have to be aggressively doing health checks and have those max-entries set very high at the same time to use the amount of memory you are seeing consumed. Your increase looked pretty steady except in a couple places, likely influx of user traffic. But a steady increase would make me think of something that was constantly running against the webseal.
Also, from what you were describing about the dynatrace extension I think that is the same one I was thinking of. Very useful info!
I just had a thought, any chance the dynatrace monitor could be pulling stats and causing a memory usage when those stats are pulled? That is one thing I was just thinking you have different that not a lot of others likely use where this would not have been seen before.
------------------------------
Matt Jenkins
Original Message:
Sent: Mon April 10, 2023 03:39 PM
From: Bipin Dash
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
Matt, Dynatrace provides this licensed agent to deploy in ISVA. You can deploy this agent in System -> extension and also need to specify API key and credential details.
Session time out is 1800 Secs. Even in week end, the user count is 0 and DSC session count is also 0, still there are spikes. During business day, the user load is pretty less, max 100 users. I did collect some pdweb.debug, threads and snoops. and didn't find any corelation, was thinking could be hgher usage of worker threads but doesn't look like that's the reason.
------------------------------
Bipin Dash
Original Message:
Sent: Mon April 10, 2023 01:11 PM
From: Matt Jenkins
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
Bipin, very nice you have it tied into dynatrace. Is that the dynatrace plugin that IBM has? I saw that availability somewhere. Very useful in this case!
That's interesting webseald would keep growing. What are your session timeouts? I am just thinking maybe something where user authentication load leaving sessions open could be consuming memory. Does it seem to grow in relationship with request and/or authentication requests where you could corelate it to perhaps the mechanism inside webseal that could be causing the issue?
------------------------------
Matt Jenkins
Original Message:
Sent: Mon April 10, 2023 11:19 AM
From: Bipin Dash
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
Thank you Matt for you response.
We don't have the AAC on the same WebSEAL virtual appliance. However we don't see any memory spike on AAC VA.
Yes, it's only for the Webseald process. It's cluster system with 2 Webseal VA and we have DSC enabled. DSC process doesn't consume more memory. Below is the screen shot from the dynatrace monitoring system.
The total memory consumption for webseald process is 3.03 GB, it keeps growing up over day by day (+200/300 MB).
We don't have any plan to go 10.0.5 for now because we have to rescope everything considering the time line.
I agree with you every environment is different. For time being, will wait for L2. I will take a look regarding the modsecurity advanced tuning parameter if it gets enabled by default.
Appreciate you response.
------------------------------
Bipin Dash
Original Message:
Sent: Mon April 10, 2023 09:14 AM
From: Matt Jenkins
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
I would take a support file when memory is high. Unzip the support.txt from the support file, and check the support.txt and see what the process list shows as the top memory consumer. Are you certain it is the reverse proxy instances (webseald) consuming the memory?
Check to see if the webseald process is consuming memory or one of the java processes on the appliance (I am assuming virtual?). Are there any other functions this appliance is performing? Does it have the AAC or federation license installed? If yes, then the runtime is likely running, so that could be the culprit. Is the appliance part of a cluster? Is it acting as a cluster master where it may be running the PD web runtime, DSC, config DB, and/or HVDB functions? In any case, the support.txt is going to be your first point of reference to determine what process is consuming the memory. You need to figure that out first unless you already know the webseald process is at fault.
If webseald, then I would start looking at things like http transforms, user mapping rules, cert mapping rules, WCP/ModSec (likely not ModSec since that was a 10.0.5.0 feature unless you are using the technical preview) configuration. ModSecurity was known to have a bad memory leak in v10.0.4.0 versions, but I am unsure if it is loaded without the advanced tuning parameter in v10.0.4.0.
Also, have you considered just upgrading to v10.0.5.0 or is that not an option for you? Good luck, I'm curious what you find out. We're fighting a memory leak with the RTSS (AAC/Federation) right now that L2 hasn't been able to figure out yet. Unfortunately the configurations are so unique in some environments it makes it pretty difficult to track things down sometimes.
------------------------------
Matt Jenkins
Original Message:
Sent: Fri April 07, 2023 11:38 AM
From: Bipin Dash
Subject: Memory Leak issue in ISVA 10.0.4.0 IF1
Hello Team,
We did an upgrade from 9.0.7.2 10.0.4 with IF 1. There is a clear indication memory spike issue on the reverse proxy instances. Created ticket, but not getting enough help from L2 & L3. L2 escalated to L3 but they are not convinced there is memory leak.
- Have done thorough analysis on reverse proxy session cache, worker thread and other tuning.
- Memory keeps growing up even there is a 0 user load in system.
I am adding the graph for last 30 days. On March 9th we did upgraded. I created a ticket after 2-3 days. but it's been close to 1 month. Am posting here if anyone can help in this community. Appreciate!
Case No - TS012447418
------------------------------
Bipin Dash
------------------------------