Hi,

My application servers hungs after 12 minutes of restart with the following messge ant the stack trace.

[10/30/13 20:06:14:863 CDT] 00000020 ThreadMonitor W   WSVR0605W: Thread "MessageListenerThreadPool : 0" (00000025) has been active for 747841 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.
        at java.lang.Thread.sleep(Native Method)
        at com.norkom.monitoring.business.dao.StateStoreDao.synchronize(StateStoreDao.java:87)
        at com.norkom.monitoring.business.scenarioserver.storage.StateConsistencyHandler.initialize(StateConsistencyHandler.java:349)
        at com.norkom.monitoring.business.scenarioserver.processing.ProcessingCore.configureStateConsistency(ProcessingCore.java:421)
        at com.norkom.monitoring.business.scenarioserver.processing.ApplicationBase.ensureStateConsistency(ApplicationBase.java:699)
        at com.norkom.monitoring.business.scenarioserver.processing.ApplicationBase.initDomains(ApplicationBase.java:524)
        at com.norkom.monitoring.business.scenarioserver.processing.ApplicationAction.initializeApplication(ApplicationAction.java:216)
        at com.norkom.monitoring.business.scenarioserver.processing.ApplicationAction.executeMessage(ApplicationAction.java:66)
        at com.norkom.monitoring.services.internal.mdb.ProcessingMdb.onMessage(ProcessingMdb.java:140)
        at com.ibm.ejs.jms.listener.MDBWrapper$PriviledgedOnMessage.run(MDBWrapper.java:302)
        at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
        at com.ibm.ejs.jms.listener.MDBWrapper.callOnMessage(MDBWrapper.java:271)
        at com.ibm.ejs.jms.listener.MDBWrapper.onMessage(MDBWrapper.java:240)
        at com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:147)
        at com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2846)
        at com.ibm.mq.jms.MQSession.run(MQSession.java:862)
        at com.ibm.ejs.jms.JMSSessionHandle.run(JMSSessionHandle.java:1055)
        at com.ibm.ejs.jms.listener.ServerSession.connectionConsumerOnMessage(ServerSession.java:1082)
        at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:752)
        at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:718)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:47)
        at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:98)
        at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:136)
        at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:574)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)

I did try to collect thread dumps and analyze it and I am seeing 2 blocked threads which I am not sure how to get rid of.

Thread Name	Norkom Base Platform Scheduler_QuartzSchedulerThread
State	Blocked
Java Stack	at java.lang.Thread.sleep(Native Method) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
Native Stack	No Native stack trace available

Thread Name	DefaultQuartzScheduler_QuartzSchedulerThread
State	Blocked
Java Stack	at java.lang.Thread.sleep(Native Method) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)
Native Stack	No Native stack trace available

Should I look into tweaking My MessageListenerThreadPool settings, they are currently at 40/40. This was working few days back and we suddenly started seeing this error, not sure if there any dead locks / where they are?

Can anybody please provide  any insight? I can provide more information if needed.

Thanks in Advance"""

Hi WASUser,

  3 weeks ago you have had similar behavior.  
  www.websphereusergroup.org/go/thread/vie...
   
  At that time the thread is freed after some time. Now?
 
  Joseph is wondering if the application is sleeping for some reason (Threads are sleep).
 
  at java.lang.Thread.sleep(Native Method)
  at com.norkom.acquisition.business.scheduler.batchbridge.BatchBridgeJob.sleep(BatchBridgeJob.java:573)
 
  at java.lang.Thread.sleep(Native Method)
  at com.norkom.monitoring.business.dao.StateStoreDao.synchronize(StateStoreDao.java:87)
 

Your threads are blocked by another thread? which thread?

quartz is used to scheduled tasks (org.quartz.core.QuartzSchedulerThread.run) 
Have you talked with application developer?
 

regards

Wow ! you are amazing remembering abt my last post.  It is the same applicaiton

The thread in the last post was recovering  itself and I know the application loads huge amount of data during that period,  which usually happens at midnight, so I was ok with that.

this time , There are 3 jvms in the cluster and all hung with the same message, I need to kill the jvm everytime and would start it back up, but after 12 minutes of time or so I  will see that hung thread which will remain for ever.

Its not an ingrown application, its a vendor app and I am working with  them too.  They want to blame it on the CPU /memory allocation but I am not seeing issues reagarding OOM /CPU starvation/CPU time slices.

Just trying to pin point the issue and I am having hard time doing that.

that looks to me like a lock issue ...
can you try running only one JVM ?

We had to delete some tables on the DB to get pass through this. Vendor is working on to prevent this.