WebSphere Application Server & Liberty

WebSphere Application Server & Liberty

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  Java Core dump analysis

    Posted Mon July 16, 2018 04:35 AM
    Edited by venkat krish Mon July 16, 2018 05:41 AM
    Hi,

    I have got the core dumps while out of memory on WAS server.

    when trying to analyze multiple cores i have seen one common thread in both cores.

    Thread Name com.ibm.son.mesh.Peer-tcp-port-11034
    State Runnable
    Monitor Owns Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110ED8
    Java Stack at java/io/ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1884(Compiled Code))
    at java/io/ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1793(Compiled Code))
    at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1501(Compiled Code))
    at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code))
    at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code))
    at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code))
    at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code))
    at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code))
    at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code))
    at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code))
    at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code))
    at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code))
    at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code))
    at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1555(Compiled Code))
    at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1516(Compiled Code))
    at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433(Compiled Code))
    at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179(Compiled Code))
    at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:346(Compiled Code))
    at com/ibm/son/util/Util.serializeHelper(Util.java:320(Compiled Code))
    at com/ibm/son/util/Util.serialize(Util.java:352(Compiled Code))
    at com/ibm/son/mesh/AbstractUDPImpl.send(AbstractUDPImpl.java:144(Compiled Code))
    at com/ibm/son/mesh/AbstractUDPImpl.procMessage(AbstractUDPImpl.java:371(Compiled Code))
    at com/ibm/son/mesh/RawUDPImpl.receive(RawUDPImpl.java:220(Compiled Code))
    at com/ibm/son/mesh/RawSelectorTaskImpl.run(RawSelectorTaskImpl.java:245(Compiled Code))
    at com/ibm/son/mesh/Peer.run(Peer.java:808)

    this was blocked because of above one.

    State Blocked
    Monitor Waiting for Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110ED8
    Java Stack at com/ibm/son/mesh/CfwTCPImpl.complete(CfwTCPImpl.java:1059(Compiled Code))
    at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
    at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
    at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
    at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
    at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code))
    at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code))
    at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
    at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1881(Compiled Code))

    State Blocked
    Monitor Waiting for Monitor Lock on com/ibm/son/mesh/Peer@0x0000000002110B88
    Java Stack at com/ibm/ws/bbson/BBFactoryImpl$MyAlarmListener.alarmWork(BBFactoryImpl.java:2078(Compiled Code))
    at com/ibm/ws/bbson/BBFactoryImpl$MyAlarmListener.alarm(BBFactoryImpl.java:1828(Compiled Code))
    at com/ibm/ejs/util/am/_Alarm.runImpl(_Alarm.java:151(Compiled Code))
    at com/ibm/ejs/util/am/_Alarm.run(_Alarm.java:136(Compiled Code))
    at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1881(Compiled Code))
    Native Stack (0x00007FB7CFB0EB62 [libj9prt26.so+0x13b62])
    (0x00007FB7CFB1CE3F [libj9prt26.so+0x21e3f])
    (0x00007FB7CFB0E8AB [libj9prt26.so+0x138ab])
    (0x00007FB7CFB0E9A7 [libj9prt26.so+0x139a7])
    (0x00007FB7CFB1CE3F [libj9prt26.so+0x21e3f])
    (0x00007FB7CFB0E4CB [libj9prt26.so+0x134cb])
    (0x00007FB7CFB08322 [libj9prt26.so+0xd322])
    (0x0000003FD300F7E0 [libpthread.so.0+0xf7e0])
    pthread_cond_wait+0xca (0x0000003FD300B68A [libpthread.so.0+0xb68a])
    (0x00007FB7CFD5A62F [libj9thr26.so+0x862f])
    (0x00007FB7CFECA439 [libj9vm26.so+0x65439])

    can some please help me to understand the above.

    ------------------------------
    venkat

    ------------------------------


  • 2.  RE: Java Core dump analysis

    Posted Tue July 17, 2018 02:21 AM
    Hello Venkat,
    are you trying to find the root cause for the OOM? If that's the case you'd need at least a heap - or even better - a system dump of the JVM.

    From the threads ( btw - what is your environment? Which WAS (tWAS or Liberty)? Version? FP? What is port 11034 used for?) it's impossible to say what causes the OOM. All we can see here is that the WAS server is sending blocked data via port 11034.

    ------------------------------
    Hermann Huebler
    ------------------------------

    Original Message


  • 3.  RE: Java Core dump analysis

    Posted Tue July 17, 2018 04:52 AM
    Hi hermann,

    thanks for the help, yes i am looking to find out root cause of OOM  and slow running threads.

    i have  seen the the above mentioned thread are running for hour(when look at two dumps taken in hour span seen the above threads are running for a while and dont have a clue on what it is doing )

    we are running WAS ND 8.5.5.10 and port which is blocked is overlay tcp listner adrress.

    ------------------------------
    venkat
    ------------------------------

    Original Message


  • 4.  RE: Java Core dump analysis

    Posted Wed July 18, 2018 02:55 AM
    Hello Venkat,
    well - I'd start with the analysis of the verbose:gc output file and see how the GC behaves. Next I'd suggest you look at th heap- /system dump using Memory Analysis Tool which is available with the IBM Support Assistent. Does you application server generate a heap-/system dump when the OOM occurs?

    I'm attaching a flow chart I've found several years ago which is to my mind still valid and guides you thru the process of oom analysis. Hope that helps you too.

    ------------------------------
    Hermann Huebler
    ------------------------------

    Original Message


Related Content