Thanks for the detailed input. This is really something new to me. Glad I posted the issue here. Would you say that temp directory will have the data even after the appliance/VM restart? Since I restarted the appliance, I believe I would have lost that data from the perticular day.
In such case , would you recommend to pull the dpmon files from CLI to somewhere else and debug?
- Prathamesh
Original Message:
Sent: Thu April 27, 2023 10:50 AM
From: Ivan Heninger
Subject: High Memory Usage in IDG datapower
The json and xsl compile events can spike memory. Things like validation schemas are "differed" compile, compiled on first execution, so they can appear long after bootstrap.
You can use the data found in default domain's temporary://dpmon/ directory to observe mem, cpu, disk and network metrics. Add the .nmon file name extension to the dpmon logs and load them into your favorite nmon data rendering tool, I like NMONVisualizer. In the "top" data you will see drouter process, this is the main drouter runtime, and you can review the ResidentSetSize(RSS).
The 9 files in dpmon directory contain about 2 hrs of history.
------------------------------
Ivan Heninger
Original Message:
Sent: Thu April 27, 2023 10:22 AM
From: Prathamesh Dixit
Subject: High Memory Usage in IDG datapower
Hi Matthias,
Thanks for your detailed inputs. There is no external monitoring in place. All we have is inside datapower in form of logs.
1. I would say memory spike up was sudden. As I had mange to login the environment successfully prior days.
2. I would not expect any heavy traffic being non-prod environment.
3. I did try to clear stylesheets during intermittent logins. However, the memory was constant to 98% still.
In raised PMR, I managed to provide IBM post restart error report (which was hardly useful to debug actual cause). However, IBM identified that in few MQ managers retry attempt was set to 0. This appears as known issue to CPU/memory spikes. I would be investigating on that part as well now.
Keeping close eye on environment. Currently memory is stable to 11%.
Thanks !
- Prathamesh
------------------------------
Prathamesh Dixit
Original Message:
Sent: Thu April 27, 2023 06:57 AM
From: Matthias Siebler
Subject: High Memory Usage in IDG datapower
Hi do you have monitoring in place? Can you see if the mem growth was slow or fast?
I would start by looking at 'expected' drivers of mem growth; was there higher traffic at this time?
Were the caches getting full or were they constant?
Any recent changes or is the system static?
and so on
------------------------------
Matthias Siebler
MA
Original Message:
Sent: Wed April 26, 2023 04:12 AM
From: Prathamesh Dixit
Subject: High Memory Usage in IDG datapower
Restart did the job. However I need to analyse why memory went so high. I had to login to VMConsole (CLI) and do "shutdown reboot 30" which restarts the whole datapower in 30 seconds. Thanks for your input.
------------------------------
Prathamesh Dixit
Original Message:
Sent: Tue April 25, 2023 06:06 PM
From: Joseph Morgan
Subject: High Memory Usage in IDG datapower
Try to reload the firmware. If that doesn't clean it up, it will require a restart.
------------------------------
Joseph Morgan
Original Message:
Sent: Tue April 25, 2023 01:44 PM
From: Prathamesh Dixit
Subject: High Memory Usage in IDG datapower
Hi,
High memory usage is observed in one of the datapower with 98%. We have ensured that all probes are disabled through CLI. We have also ensured that domains are restarted and caches are cleared. However, the memory utilization still stays at 98%. We are also unable to login to Web GUI. It is login intermittently and GUI is behaving unstable. Thus unable to generate any error report or backups.
Can you please suggest how this issue can be resolved? The memory we have is 16gb. I would like to get the memory usage to stable condition along with access to web GUI.
Thanks !
------------------------------
Prathamesh Dixit
------------------------------