AIOps

AIOps

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.


#ITAutomation
#AIOps
#CloudPakforAIOps
#AIOps

 View Only

  • 1.  WXS hung threads - FDC - depth limit reached

    Posted Fri August 23, 2013 01:37 PM
    I am using WXS standalone 8.6.0.3. I am testing in a lab environment with a simple 1 grid setup that is using 1 primary and 1 sync replicas. Currently no clients as connected and/or sending data to the grid


    I am having performance problems when teardowning/starting back up container JVMs in zones which end up hanging threads.

    This has caused me to start looking into some of the errors I see in FDCs when starting catalog/container servers.


    1st exceptions is depth limit reached:

    [8/23/13 10:05:00:669 CDT]     FFDC Exception:com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException SourceId:com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler.runXIOQueryRevisionPeerMode ProbeId:519 Reporter:com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler@1738bebf


    com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException [originating=204.53.93.118:4009;causedby=204.53.93.118:4009;exid=0]: com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException


            at RemovedByServerSerialization.RemovedByServerSerialization(RemovedByServerSerialization:0)


    Caused by: java.util.concurrent.RejectedExecutionException: java.util.concurrent.RejectedExecutionException


            ... 1 more


     


    Reporter BEGIN:com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler@1738bebf


    com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler::objectGrid BEGIN:com.ibm.ws.objectgrid.ObjectGridImpl@3c4c463f


     com.ibm.ws.objectgrid.ObjectGridImpl::CLASS_NAME:com.ibm.ws.objectgrid.ObjectGridImpl


     com.ibm.ws.objectgrid.ObjectGridImpl::pendingDynamicMaps BEGIN:java.lang.ThreadLocal@510beba5


      java.lang.ThreadLocal::threadLocalHashCode:-1065312117


       java.lang.ThreadLocal::nextHashCode:java.util.concurrent.atomic.AtomicInteger@b97df067 depth limit reached


    2nd exception is:

     Index  Count  Time of first Occurrence    Time of last Occurrence     Exception SourceId ProbeId
    ------+------+---------------------------+---------------------------+---------------------------
         0     44    8/23/13 10:05:00:598 CDT    8/23/13 10:06:01:342 CDT com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler.runXIOQueryRevisionPeerMode 519 /qual/86/logs/ffdc/86conq3Ldap_acd6f801_13.08.23_10.05.00.6578660052580514828139.txt
         1     43    8/23/13 10:05:01:136 CDT    8/23/13 10:05:56:979 CDT com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler.run 172 /qual/86/logs/ffdc/86conq3Ldap_acd6f801_13.08.23_10.05.01.1361442322238434409130.txt



    More details on:


    DC Exception:com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException SourceId:com.ibm.ws.objectgrid.replication.SynchronousRe
    plicaRevisionShardImpl$RevisionPeerModeHandler.runXIOQueryRevisionPeerMode ProbeId:519 Reporter:com.ibm.ws.objectgrid.replication.SynchronousReplicaRevision
    ShardImpl$RevisionPeerModeHandler@1738bebf
    com.ibm.ws.xsspi.xio.exception.ObjectGridXIOException [originating=204.53.93.118:4009;causedby=204.53.93.118:4009;exid=0]: com.ibm.ws.xsspi.xio.exception.Ob
    jectGridXIOException
            at RemovedByServerSerialization.RemovedByServerSerialization(RemovedByServerSerialization:0)
    Caused by: java.util.concurrent.RejectedExecutionException: java.util.concurrent.RejectedExecutionException
            ... 1 more


    Reporter BEGIN:com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler@1738bebf
    com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl$RevisionPeerModeHandler::objectGrid BEGIN:com.ibm.ws.objectgrid.ObjectGridImpl@3c4c463
    f
     com.ibm.ws.objectgrid.ObjectGridImpl::CLASS_NAME:com.ibm.ws.objectgrid.ObjectGridImpl
     com.ibm.ws.objectgrid.ObjectGridImpl::pendingDynamicMaps BEGIN:java.lang.ThreadLocal@510beba5
      java.lang.ThreadLocal::threadLocalHashCode:-1065312117
       java.lang.ThreadLocal::nextHashCode:java.util.concurrent.atomic.AtomicInteger@b97df067 depth limit reached
      java.lang.ThreadLocal::HASH_INCREMENT:1640531527
      END:java.lang.ThreadLocal@510beba5


    Any suggestions on how to resolve this depth limit exception?

    Thanks!

    Brett



  • 2.  WXS hung threads - FDC - depth limit reached

    Posted Thu October 10, 2013 07:03 PM
    This is now fixed!

    I was using a custom memory evictor. This evictor was working fine with WXS8.5 but was destroying threads during the restart process on WXS8.6.  Since fixing the custom memory evictor my hung thread issue has been resolved.


Related Content