Hi,

I have got the core dumps while out of memory on WAS server.

when trying to analyze multiple cores i have seen one common thread in both cores.

Thread Name	com.ibm.son.mesh.Peer-tcp-port-11034
State	Runnable
Monitor	Owns Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110ED8
Java Stack	at java/io/ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1884(Compiled Code)) at java/io/ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1793(Compiled Code)) at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1501(Compiled Code)) at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code)) at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code)) at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code)) at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code)) at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code)) at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code)) at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code)) at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code)) at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code)) at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code)) at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code)) at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code)) at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code)) at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code)) at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:346(Compiled Code)) at com/ibm/son/util/Util.serializeHelper(Util.java:320(Compiled Code)) at com/ibm/son/util/Util.serialize(Util.java:352(Compiled Code)) at com/ibm/son/mesh/AbstractUDPImpl.send(AbstractUDPImpl.java:144(Compiled Code)) at com/ibm/son/mesh/AbstractUDPImpl.procMessage(AbstractUDPImpl.java:371(Compiled Code)) at com/ibm/son/mesh/RawUDPImpl.receive(RawUDPImpl.java:220(Compiled Code)) at com/ibm/son/mesh/RawSelectorTaskImpl.run(RawSelectorTaskImpl.java:245(Compiled Code)) at com/ibm/son/mesh/Peer.run(Peer.java:808)

this was blocked because of above one.

State	Blocked
Monitor	Waiting for Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110ED8
Java Stack	at com/ibm/son/mesh/CfwTCPImpl.complete(CfwTCPImpl.java:1059(Compiled Code)) at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code)) at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code)) at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code)) at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code)) at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code)) at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code)) at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code)) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1881(Compiled Code))

State	Blocked
Monitor	Waiting for Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110B88
Java Stack	at com/ibm/ws/bbson/BBFactoryImpl$MyAlarmListener.alarmWork(BBFactoryImpl.java:2078(Compiled Code)) at com/ibm/ws/bbson/BBFactoryImpl$MyAlarmListener.alarm(BBFactoryImpl.java:1828(Compiled Code)) at com/ibm/ejs/util/am/_Alarm.runImpl(_Alarm.java:151(Compiled Code)) at com/ibm/ejs/util/am/_Alarm.run(_Alarm.java:136(Compiled Code)) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1881(Compiled Code))
Native Stack	(0x00007FB7CFB0EB62 [libj9prt26.so+0x13b62]) (0x00007FB7CFB1CE3F [libj9prt26.so+0x21e3f]) (0x00007FB7CFB0E8AB [libj9prt26.so+0x138ab]) (0x00007FB7CFB0E9A7 [libj9prt26.so+0x139a7]) (0x00007FB7CFB1CE3F [libj9prt26.so+0x21e3f]) (0x00007FB7CFB0E4CB [libj9prt26.so+0x134cb]) (0x00007FB7CFB08322 [libj9prt26.so+0xd322]) (0x0000003FD300F7E0 [libpthread.so.0+0xf7e0]) pthread_cond_wait+0xca (0x0000003FD300B68A [libpthread.so.0+0xb68a]) (0x00007FB7CFD5A62F [libj9thr26.so+0x862f]) (0x00007FB7CFECA439 [libj9vm26.so+0x65439])

can some please help me to understand the above.

------------------------------
venkat

------------------------------

Hello Venkat,
are you trying to find the root cause for the OOM? If that's the case you'd need at least a heap - or even better - a system dump of the JVM.

From the threads ( btw - what is your environment? Which WAS (tWAS or Liberty)? Version? FP? What is port 11034 used for?) it's impossible to say what causes the OOM. All we can see here is that the WAS server is sending blocked data via port 11034.

------------------------------
Hermann Huebler
------------------------------

Hi hermann,

thanks for the help, yes i am looking to find out root cause of OOM  and slow running threads.

i have  seen the the above mentioned thread are running for hour(when look at two dumps taken in hour span seen the above threads are running for a while and dont have a clue on what it is doing )

we are running WAS ND 8.5.5.10 and port which is blocked is overlay tcp listner adrress.

------------------------------
venkat
------------------------------

Hello Venkat,
well - I'd start with the analysis of the verbose:gc output file and see how the GC behaves. Next I'd suggest you look at th heap- /system dump using Memory Analysis Tool which is available with the IBM Support Assistent. Does you application server generate a heap-/system dump when the OOM occurs?

I'm attaching a flow chart I've found several years ago which is to my mind still valid and guides you thru the process of oom analysis. Hope that helps you too.

------------------------------
Hermann Huebler
------------------------------

Original Message:
Sent: 07-17-2018 04:51
From: venkat krish
Subject: Java Core dump analysis

Hi hermann,

thanks for the help, yes i am looking to find out root cause of OOM  and slow running threads.

i have  seen the the above mentioned thread are running for hour(when look at two dumps taken in hour span seen the above threads are running for a while and dont have a clue on what it is doing )

we are running WAS ND 8.5.5.10 and port which is blocked is overlay tcp listner adrress.

------------------------------
venkat
------------------------------