After increased that java heap size, restarted the ILMT server service, it seems to fix the issue. At that time when the error occurred, someone was doing the part number import. Which I don't think that would cause the issue. Anyway, will continue to monitor.
Original Message:
Sent: Tue February 27, 2024 09:05 AM
From: Oktawian Powązka
Subject: ILMT Java Core Dump - Out of Memory error.
Forgot to ask...
Are those OutOfMemoryError exceptions correlated with any particular events like Import, Audit snapshots generation, etc. ?
------------------------------
Thank you,
Oktawian
Oktawian Powązka, L3 Support
IBM License Metric Tool
Original Message:
Sent: Tue February 27, 2024 01:30 AM
From: Oktawian Powązka
Subject: ILMT Java Core Dump - Out of Memory error.
Hello York,
All those errors are referencing just one problem : shortage of Java heap size.
The only place to increase the Java heap is through server runtime memory (Xmx).
That warning message (The License Metric Tool server is running out of memory...) shows up when occupied heap goes over 90% of available space.
If increasing the runtime memory (Xmx) doesn't help pls. enable the logging of memory consumption feature :
https://www.ibm.com/docs/en/license-metric-tool?topic=file-enabling-logging-memory-consumption
------------------------------
Thank you,
Oktawian
Oktawian Powązka, L3 Support
IBM License Metric Tool
Original Message:
Sent: Mon February 26, 2024 04:14 PM
From: York Ly
Subject: ILMT Java Core Dump - Out of Memory error.
Hi Team, all of the sudden ILMT is facing out of memory error in the last few days. it was been working well since. Found errors in these logs below.
I have increased the server memory size to 6144m as per doc link below. Is this the only place which those errors are referenced to?
https://www.ibm.com/docs/en/license-metric-tool?topic=environments-tuning-server-memory-size
Please advise.
===================================
Tema.log:
[2/26/24 10:35:40:999 UTC] 00000cdc com.ibm.ws.webcontainer.webapp I SRVE0292I: Servlet Message - [tema]:.[INFO] Started POST "/api/sam/etl/launch" for 172.25.135.121 at 2024-02-26 10:35:40 +0000
[2/26/24 10:35:40:999 UTC] 00000cdc com.ibm.ws.webcontainer.webapp I SRVE0292I: Servlet Message - [tema]:.[INFO] Processing by SAM::LmtRestAPIController#etl_launch as HTML
[2/26/24 10:46:00:429 UTC] 00000be8 com.ibm.ws.webcontainer.webapp I SRVE0292I: Servlet Message - [tema]:.[WARN] The License Metric Tool server is running out of memory and may not be able to process the data in a timely manner. Delays can be expected. To solve the problem, increase the runtime memory size on the License Metric Tool server.
Console.log
Launching server1 (WebSphere Application Server 23.0.0.6/wlp-1.0.78.cl230620230612-1100) on IBM J9 VM, version 8.0.8.5 - pwa6480sr8fp5-20230418_01(SR8 FP5) (en_US)
[AUDIT ] CWWKE0001I: The server server1 has been launched.
JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2024/02/26 05:48:51 - please wait.
JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2024/02/26 05:48:51 - please wait.
JVMDUMP032I JVM requested Java dump using 'C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054851.8964.0001.txt' in response to an event
JVMDUMP010I Java dump written to C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054851.8964.0001.txt
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2024/02/26 05:49:00 - please wait.
JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2024/02/26 05:49:00 - please wait.
JVMDUMP032I JVM requested Java dump using 'C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054851.8964.0002.txt' in response to an event
JVMDUMP010I Java dump written to C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054851.8964.0002.txt
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Java dump using 'C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054900.8964.0004.txt' in response to an event
JVMDUMP010I Java dump written to C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054900.8964.0004.txt
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2024/02/26 05:49:02 - please wait.
JVMDUMP032I JVM requested Java dump using 'C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054900.8964.0003.txt' in response to an event
Java core log
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TICHARSET 1252
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
1TIDATETIMEUTC Date: 2024/02/26 at 10:49:01:636 (UTC)
1TIDATETIME Date: 2024/02/26 at 05:49:01:636
1TITIMEZONE Timezone: (unavailable)
1TINANOTIME System nanotime: 87733839798930
1TIFILENAME Javacore filename: C:\Program Files\ibm\LMT\wlp\usr\servers\server1\logs/javacore.20240226.054900.8964.0004.txt
1TIREQFLAGS Request Flags: 0x41 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x84 (exclusive_vm_access+trace_disabled)
NULL ------------------------------------------------------------------------
------------------------------
York Ly
------------------------------