Hello,

I am running my application on WAS installed on AIX & connected to DB which is oracle, when try to access the application URL it gives vey poor response & I would see error saying it cant connect to DB (not able to get the thread to connect to DB) after sometime as excepted the Webscontainer thread gets hunged.

But the surprising point is other application pointing to same DB running on the same server works fine, even the problematic application after all hung thread get realeased it works fine.

I have consulted my DBA who says all is OK from there end (which they always say :) ). Also tried killing the JVM & restarting it but no luck.

My concern is this problem is very intermittent & I tried checking all the configuration as well, This morning 12:00 PM IST now its working fine again. Cant understand where is the actual issue and what is causing it.

Should I check with application team (developers) once?

Thanks
Sumit Tambe

Hi Sumit,

  Check with your DBA the number of sessions(PROCESSES & SESSION) allowed to DB.
 
  f.e if you have two datasources with max 10 sessions defined but your DB is configured to only 10 (not 20) if one of the application has 10 connections established the other datasource can have problems to get connections.
 
  in the same way if you have only one datasource used by both applications but the max sessions if lower than number of sessions allowed to DB if one application takes all of them the other can have problems

  Have you check stopping one of the both applications? can you put the error?
 
  hope this helps.
 
Regards

Hello Gabriel,

Thanks for the valueable comment, its very logical.

But my application uses separate Data-sources to connect to the same DB, so they are using different pool.

For now I have asked my DB the "maximum connections allowed to the DB instance", also I guess it can be due to rigours testing performed so I have asked my testing team as well.

But the later point can be neglected since I was getting the same issue on Sunday when there as no testing performed.

The error i am getting is as below

[9/25/13 12:07:32:943 CEST] 00000021 FreePool      E   J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/DS_CRPE_T.
[9/25/13 12:07:32:965 CEST] 00000021 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[9/25/13 12:07:32:971 CEST] 00000021 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 180000

These results in many webcontainer hung threads message as below

[9/24/13 17:31:52:412 CEST] 00000057 ThreadMonitor W   WSVR0606W: Thread "WebContainer : 42" (00000057) was previously reported to be hung but has completed.  It was active for approximately 1572481 milliseconds.  There is/are 11 thread(s) in total in the server that still may be hung.

[9/24/13 17:31:52:414 CEST] 0000003a ThreadMonitor W   WSVR0606W: Thread "WebContainer : 26" (0000003a) was previously reported to be hung but has completed.  It was active for approximately 10630659 milliseconds.  There is/are 10 thread(s) in total in the server that still may be hung.

On release of all the hung threads the application becomes accessible normal.

Today early morning it was working fine, then mid-day it stuck again to be back to normal now.

Thanks
Sumit Tambe

Below are very generic questions that you can for a JDBC connection issues, hung thereads. Interistingly, in your case threads are finishing after long time. however it is good to diagnose the issue with help of below pointers.

1. did you managed to get a thread dump when there are many hung threads? That would tell you where the threads are waiting.

2. Even if your DBA didn't agree with what you suggest... it worth asking few questions :-)
-What are the maximum sessions configured for the user you are connecting.
-When there are many hung therads in WAS, is DBA seeing all sessions from WAS and are they active or inactive
-Can DBA get what the session is doing (easy if they use toad)

3. After the thread is released is the application transaction successful or failing?

4. What are the timeout setting on your JDBC connection

5. While you are doing the testing .. how are the application response times, are they slow ?

6. Is your application closing all the DB connections after it completed what it supposed to do ? ( you can check this in database inactive sessions)

7. Enable TPV for JDBC connection and monitor how many connections are being opened and closed.

Hello Joseph,

Thanks for the advice & guidance.

I am asked DBA the relevant question.

We tried restarting the DB instance & JVM processess but still no luck.

Also I have taken the thread dump will analyse now & update you.

Thanks
Sumit Tambe

Hello Joseph/Gabriel,

I analysed the Thread dump and as expected found a Webcontainer thread waiting on condition and owning a monitor lock on

java/util/HashMap@07000000004A3D90/07000000004A3DA8

with a Java stack as below

at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:231(Compiled Code)) at com/ibm/ejs/j2c/FreePool.queueRequest(FreePool.java:397) at com/ibm/ejs/j2c/FreePool.createOrWaitForConnection(FreePool.java:1229) at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:2391) at com/ibm/ejs/j2c/ConnectionManager.allocateMCWrapper(ConnectionManager.java:940) at com/ibm/ejs/j2c/ConnectionManager.allocateConnection(ConnectionManager.java:614) at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:449) at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:418) at org/springframework/jdbc/datasource/DataSourceUtils.doGetConnection(DataSourceUtils.java:113) at org/springframework/jdbc/datasource/DataSourceUtils.getConnection(DataSourceUtils.java:79) at org/springframework/jdbc/core/JdbcTemplate.execute(JdbcTemplate.java:577) at org/springframework/jdbc/core/JdbcTemplate.query(JdbcTemplate.java:641) at org/springframework/jdbc/core/JdbcTemplate.query(JdbcTemplate.java:670(Compiled Code)) at org/springframework/jdbc/core/JdbcTemplate.query(JdbcTemplate.java:678(Compiled Code)) at org/springframework/jdbc/core/JdbcTemplate.queryForRowSet(JdbcTemplate.java:784(Compiled Code)) at cds/fmwk/locale/JDBCExtendedPropertiesLoader.loadProperties4Locale(JDBCExtendedPropertiesLoader.java:75(Compiled Code)) at cds/fmwk/locale/JDBCExtendedPropertiesLoader.loadProperties(JDBCExtendedPropertiesLoader.java:69) at cds/fmwk/locale/ExtendedReloadableResourceBundleMessageSource.loadExtendedProperties(ExtendedReloadableResourceBundleMessageSource.java:119) at cds/fmwk/locale/ExtendedReloadableResourceBundleMessageSource.refreshPropertiesExtended(ExtendedReloadableResourceBundleMessageSource.java:90) at cds/fmwk/locale/ExtendedReloadableResourceBundleMessageSource.getProperties(ExtendedReloadableResourceBundleMessageSource.java:77(Compiled Code)) at org/springframework/context/support/ReloadableResourceBundleMessageSource.resolveCodeWithoutArguments(ReloadableResourceBundleMessageSource.java:277) at org/springframework/context/support/AbstractMessageSource.getMessageInternal(AbstractMessageSource.java:193) at org/springframework/context/support/AbstractMessageSource.getMessage(AbstractMessageSource.java:114) at org/springframework/context/support/DelegatingMessageSource.getMessage(DelegatingMessageSource.java:53) at org/springframework/context/support/DelegatingMessageSource.getMessage(DelegatingMessageSource.java:53) at org/springframework/context/support/AbstractApplicationContext.getMessage(AbstractApplicationContext.java:979) at cds/fmwk/web/struts/CustomMessageResources.getMessage(CustomMessageResources.java:50) at cds/fmwk/web/struts/CustomMessageResources.getMessage(CustomMessageResources.java:61) at org/apache/struts/taglib/TagUtils.message(TagUtils.java:1046) at org/apache/struts/taglib/bean/MessageTag.doStartTag(MessageTag.java:224) at cds/fmwk/web/struts/taglib/bean/MessageTag.doStartTag(MessageTag.java:43) at com/ibm/_jsp/_erreur._jspx_meth_bean_message_0(_erreur.java:517) at com/ibm/_jsp/_erreur._jspx_meth_fmwk$1str_replaceIgnoreCase_1(_erreur.java:544) at com/ibm/_jsp/_erreur._jspx_meth_fmwk$1str_replaceIgnoreCase_0(_erreur.java:576) at com/ibm/_jsp/_erreur._jspService(_erreur.java:216) at com/ibm/ws/jsp/runtime/HttpJspBase.service(HttpJspBase.java:88) at javax/servlet/http/HttpServlet.service(HttpServlet.java:856) at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1213) at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1154) at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:145) at com/total/rm/cardpro/web/filter/ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:60) at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:130) at com/ibm/ws/webcontainer/filter/WebAppFilterChain._doFilter(WebAppFilterChain.java:87) at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:848) at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:691) at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:654(Compiled Code)) at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:526) at com/ibm/wsspi/webcontainer/servlet/GenericServletWrapper.handleRequest(GenericServletWrapper.java:122) at com/ibm/ws/jsp/webcontainerext/AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:248) at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:325) at com/ibm/ws/webcontainer/webapp/WebApp.sendError(WebApp.java:3049) at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:538) at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3574) at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:269) at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:831) at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1478) at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:133) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:457) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:300) at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:102) at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:136(Compiled Code)) at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:196) at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:751) at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:881) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1551)

Causing other webcontainer threads to get blocked for the same resource.

Do I need to look specifically for something in the thread dump, till now I am not able to figure out the issue.

Thanks
Sumit Tambe

_{at java/lang/Object.wait(Object.java:231(Compiled Code))}

_{at com/ibm/ejs/j2c/FreePool.queueRequest(FreePool.java:397)}

_{at com/ibm/ejs/j2c/FreePool.createOrWaitForConnection(FreePool.java:1229)}

_{at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:2391)}

What are your connection pool settings?

Hello Joseph,

We restarted the DB instance followed by JVM restart, just got the confirmation form the application team that the issue no more exists.

As we were rightly discussing the issue was with DB instance only I guess.

MY connection pool setting were proper also there were no changes to the environment from our end.

So the issue is fixed now, thanks for the help.

Trying to fix the thread dump issue now, started a different thread for it.

Thanks
Sumit Tambe