WebSphere Application Server & Liberty

WebSphere Application Server & Liberty

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Back to discussions
Expand all | Collapse all

Hung Threads (DB2 9.5 and WAS 6.1.0.25)

  • 1.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 10:38 AM
    Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    --------------------------------

    Folks,

    We keep getting hung threads in our basic Java application.
    Below is the error listed from the SystemOut.log file:

    SystemOut_09.08.18_08.19.51.log:[8/17/09 11:58:25:176 EDT] 00000024 ThreadMonitor W WSVR0605W: Thread /"Default : 107/" (00000165) has been active for 635156 milliseconds and may be hung. There is/are 3 thread(s) in total in the server that may be hung.

    And we received this message in our DB2 db2diag.log:

    2009-08-18-08.38.32.381037-240 E6797975A812 LEVEL: Error (OS)
    PID : 2026 TID : 2199133890896PROC : db2hmon
    INSTANCE: db2inst1 NODE : 000
    FUNCTION: DB2 UDB, oper system services, sqloPdbConnectSocket, probe:20
    MESSAGE : ZRC=0x810F0019=-2129723367=SQLO_CONN_REFUSED /"Connection refused/"
    CALLED : OS, -, connect
    OSERR : ECONNREFUSED (111) /"Connection refused/"
    DATA #1 : String, 32 bytes
    Unable to connect socket on path
    DATA #2 : socket, 4 bytes
    6
    DATA #3 : sockaddr, 36 bytes
    0x0000020006977630 : 0001 2F68 6F6D 652F 6461 7375 7372 312F /home/dasusr1/
    0x0000020006977640 : 6461 732F 746D 702F 6461 736E 616D 6564 das/tmp/dasnamed
    0x0000020006977650 : 7069 7065 pipe
    DATA #4 : File name, 34 bytes
    /home/dasusr1/das/tmp/dasnamedpipe

    Anyone have the same issue?

    Thank you

    Work4fun

    --------------------------------

    Posted By: work4fun at Aug 19 2009 6:36PM


  • 2.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 10:46 AM
    Are you connecting via a JDBC within WAS?
    Is your WAS a standalone or ND?

    --------------------------------

    Posted By: TGIBONEY at Aug 19 2009 8:41PM


  • 3.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 10:57 AM
    Update

    --------------------------------

    UPDATE

    Yes, we are connecting via JDBC and it is a standalone setup. No ND.
    The app is on one Linux WAS app. server and it's connected to
    three Linux DB2 UDB databases. The app is very simple. It just
    supplies a web front end to a bunch of SQL data.

    We have done the javacore thing and I've cut and pasted some of
    the errors for you to view. It looks like the threads are stuck
    in socketRead. Our application response time is usually very good until
    the problem occurs. And from our network scans, it doesn't look like it is
    from heavy traffic. We did also check our DB2 logs for errors but there were none.

    Thank you for any insight.

    /"Default : 12/" (TID:0x014A3B00, sys_thread_t:0x018AB9A8, state:CW, native ID:0x00002653) prio=5
    /"Default : 22/" (TID:0x016DC300, sys_thread_t:0x020D5880, state:CW, native ID:0x000028A6) prio=5
    /"Default : 13/" (TID:0x016DC700, sys_thread_t:0x018AB718, state:CW, native ID:0x00002654) prio=5
    /"Default : 15/" (TID:0x01CACC00, sys_thread_t:0x0149E6B0, state:R, native ID:0x00002783) prio=5
    /"Default : 14/" (TID:0x01CAD000, sys_thread_t:0x018ABC38, state:CW, native ID:0x00002782) prio=5
    /"Default : 17/" (TID:0x01CEC000, sys_thread_t:0x01C19118, state:R, native ID:0x00002785) prio=5
    /"Default : 11/" (TID:0x01CEC400, sys_thread_t:0x018AAF68, state:R, native ID:0x00002652) prio=5
    /"Default : 1/" (TID:0x021F1100, sys_thread_t:0x02164990, state:R, native ID:0x0000263A) prio=5
    /"Default : 2/" (TID:0x021F1500, sys_thread_t:0x02252728, state:R, native ID:0x0000263B) prio=5
    /"Default : 3/" (TID:0x022F4800, sys_thread_t:0x022529B8, state:R, native ID:0x0000263C) prio=5
    /"Default : 4/" (TID:0x022F4C00, sys_thread_t:0x02252C48, state:R, native ID:0x0000263D) prio=5
    /"Default : 5/" (TID:0x022F5000, sys_thread_t:0x02252ED8, state:CW, native ID:0x0000263E) prio=5
    /"Default : 6/" (TID:0x0236C100, sys_thread_t:0x02253168, state:R, native ID:0x0000263F) prio=5
    /"Default : 7/" (TID:0x0236C500, sys_thread_t:0x022533F8, state:R, native ID:0x00002640) prio=5
    /"Default : 8/" (TID:0x0236C900, sys_thread_t:0x02374F40, state:R, native ID:0x00002641) prio=5
    /"Default : 10/" (TID:0x02378500, sys_thread_t:0x02375460, state:R, native ID:0x00002643) prio=5
    /"Default : 18/" (TID:0x02ACCE00, sys_thread_t:0x01C186D8, state:R, native ID:0x00002786) prio=5
    /"Default : 19/" (TID:0x02CE1E00, sys_thread_t:0x020D5B10, state:CW, native ID:0x00002787) prio=5
    /"Default : 20/" (TID:0x02CE2200, sys_thread_t:0x015D8360, state:R, native ID:0x000028A4) prio=5
    /"Default : 21/" (TID:0x02CE2600, sys_thread_t:0x01C18E88, state:R, native ID:0x000028A5) prio=5

    --------------------------------

    Posted By: work4fun at Aug 21 2009 6:41PM


  • 4.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 10:57 AM
    You need to get a javacore when you get these msgs so you can see what the thread is doing.

    --------------------------------

    Posted By: cathy at Aug 19 2009 10:06PM


  • 5.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:02 AM
    What is the stack for the threads that are reported as hung?

    --------------------------------

    Posted By: cathy at Aug 21 2009 9:40PM


  • 6.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:04 AM
    Update

    --------------------------------

    UPDATE

    I think this is the stack for thread #12. This is from the latest javacore.

    3LKWAITNOTIFY /"Default : 12/" (0x014A3B00)
    3LKWAITNOTIFY /"Default : 14/" (0x01CAD000)
    3LKWAITNOTIFY /"Default : 19/" (0x02CE1E00)
    2LKMONINUSE sys_mon_t:0x0234E2D0 infl_mon_t: 0x0234E310:
    3LKMONOBJECT java/util/TaskQueue@11C7DC60/11C7DC6C:
    3LKNOTIFYQ Waiting to be notified:
    ................................
    4XESTACKTRACE at java/lang/Thread.run(Thread.java:810)
    3XMTHREADINFO /"Timer-5/" (TID:0x013E8400, sys_thread_t:0x0149E420, state:CW, native ID:0x0000260A) prio=5
    4XESTACKTRACE at java/lang/Object.wait(Native Method)
    4XESTACKTRACE at java/lang/Object.wait(Object.java:231(Compiled Code))
    4XESTACKTRACE at java/util/TimerThread.mainLoop(Timer.java:534)
    4XESTACKTRACE at java/util/TimerThread.run(Timer.java:487)
    3XMTHREADINFO /"Default : 12/" (TID:0x014A3B00, sys_thread_t:0x018AB9A8, state:CW, native ID:0x00002653) prio=5
    4XESTACKTRACE at java/lang/Object.wait(Native Method)

    --------------------------------

    Posted By: work4fun at Aug 24 2009 12:46PM


  • 7.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:04 AM
    Update

    --------------------------------

    UPDATE

    This is the error that is written to SystemErr.log when the problem happens.

    [8/23/09 8:17:53:364 EDT] 00000148 SystemErr R at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R com.ibm.db2.jcc.a.SqlException: An unexpected token /"END-OF-STATEMENT/" was found following /"asc , 3 , 2)AS TEMP2/". Expected tokens may include: /")/".
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.jd.e(jd.java:1664)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.jd.a(jd.java:1240)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.b.fb.h(fb.java:149)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.b.fb.a(fb.java:43)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.b.s.a(s.java:30)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.b.wb.g(wb.java:152)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.jd.n(jd.java:1220)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.kd.fb(kd.java:1779)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.kd.a(kd.java:2253)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.kd.W(kd.java:505)
    [8/23/09 8:22:28:531 EDT] 00000140 SystemErr R at com.ibm.db2.jcc.a.kd.executeQuery(kd.java:488)

    --------------------------------

    Posted By: work4fun at Aug 24 2009 12:48PM


  • 8.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:05 AM
    We had the same problem and fixed it by installing the DB2 9.5 drivers and changing the WebSphere Variables for the JDBC Driver paths to the new drivers.

    --------------------------------

    Posted By: TGIBONEY at Aug 25 2009 6:21PM


  • 9.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:06 AM
    The lines from the javacore you posted seem to have been taken from the monitor section rather than the thread details section, which with have the stack for the thread.

    It looks like the thread is waiting for a lock/monitor. Can you paste more of that section and also the actual stack from the thread detail sections for the thread that was reported as being hung?

    --------------------------------

    Posted By: cathy at Aug 25 2009 8:11PM


  • 10.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:06 AM
    Update

    --------------------------------

    UPDATE

    Kathy the dump file isn't that big but to big to paste here and I'm
    not knowledgeable enough to find the parts in the file you request.

    Will you send me an email address and I'll send you the file?

    my email is:

    dkk4work@gmail.com

    Thank you

    --------------------------------

    Posted By: work4fun at Aug 28 2009 1:23PM


  • 11.  Hung Threads (DB2 9.5 and WAS 6.1.0.25)

    Posted Wed August 11, 2010 11:07 AM
    Hello -

    Send it to cdhrtp@yahoo.com

    I'll take a look

    --------------------------------

    Posted By: cathy at Aug 28 2009 9:17PM