Hello Techies,

I got a roblem with hung Threads on IS
Cancel and killing threads didn’t make any changes
and IS memory usage raises as the hung threads stall and raised up

is there any suggestion or recomended solutions to prevent these hung threads
coz, restart server is not the best way that we can do since this case happened in Production server.
Please help.

Thanks in advance.

Could you share more details:

• version of the Integration Server
• Is this across the board for or specific to few services?
• Details of what the service that is getting hung is trying to do will be probably the first place to look i.e. is there a contention on the resource?

Hello Jery,
we use IS 10.3 and there are several project running in these server
the hung thread specificly happned in one JDBC
the other JDBC didn’t run into the same problem (normally works as well)

we try to find service that couse this, but we have no clue at all
the service which running slow were randomly appears each time the hung happen

Hi @Tegar_ASK

You can check the JDBC related information such as connectivity to the database - when the issue happens is the node reachable , is there a timeout after which instead of waiting response and being blocked may be will just terminate and return.

Thread dumps are of course the best way to go.

-NP

Hi @Tegar_ASK,
Since you have mentioned that the issue is happening for one specific JDBC adapter service, the next time this occurs you could run the service
pub.art.connection:getConnectionStatistics
this will give you an idea of the usage of the connection pool and its health, are there available connections or not.
You might also want to check with the developers if they are using a JDBC "Local Transaction"or "XA Transaction"as these too could result in long waits for locks to be released on the database (I personally stay away from using transactional connections unless absolutely required).
Also check with the development team what is it that they are are trying to do with the flow and if it has been tested in non-production environments with similar data volume.

You should also be able to identifiy the service that is taking long from the service usage page.
HTH
Regards
~Jerry

wow this is new things to me “pub.art.connection:getConnectionStatistics”
i’ll try to look up this service.
oh , is there any Transaction Type recomendation for APIs in which, the API traffic is massive and also db adapter using “stored procedure with signature” type?

Best regards

Hi @Tegar_ASK ,
The transaction type for the JDBC Adapter connection should reflect the type of operation its going to be used for please review the details of the “Transaction Types” along with "Transaction Isolation Level Settings"in the webMethods Adapter for JDBC Installation and User’s Guide 10.3 guide to get a better understanding of the transaction type you need to use. If you have no need for a "Local Transaction"connection I would strongly encourage that use a “No_Transaction” connection as much as possible.
HTH
~Jerry

thx Jerry for the advise
but we allready using the recomendation settup for jdbc (based on Guide)
and we still have the same issue…

is there any misscofiguration on our jdbc?

BestRegards

Stability - Is your database connection and/or network stable? Increase the logging level for JDBC Adapter in the IS, and look in the server logs.

Locking - Thread dumps will show you if there is a locked thread, which is causing other requests to wait endlessly. These are Java thread dumps and there’s good content online to understand how to interpret them.

Driver Properties - Test these settings in the JDBC connection on a lower environment - driverType=thin;connectionProperties={oracle.net.CONNECT_TIMEOUT=10000,oracle.jdbc.ReadTimeout=60000}

Property Info -

1. ReadTimeout - oracle.jdbc.ReadTimeout
2. CONNECT_TIMEOUT - OracleConnection (Oracle Database JDBC Java API Reference)

KM

waow thank you Kasi for the details
Im gonna try it out
hope it gets better with your recomendation

I will post the result using this setup
as soon as possible

Best Regards

I second that recommendation. And would also note that “No Transaction” type does not mean there is no transaction – it just means the adapter is not explicitly managing the transaction.

thx Nagendra,
is there any documentation that clearly describe how to read thread dumps
i’ll try to analyze our thread dumps
best regrads.

Hi,

in addition to jerry´s questions:
Can you take and analyse some thread dumps on the JVM IS is running in?
This might help to identify which threads get stuck and why.

Regards,
Holger

thx for the advice Tomsen,
ill try to analyze the thead dumps
but
is there any action that we can do, aside from restarting the IS server?
or maybe we can put timeout setting for the thread so that it wont be hung

thx for the advice Tomsen,
ill try to analyze the thead dumps
but
is there any action that we can do, aside from restarting the IS server?
or maybe we can put timeout setting for the thread so that it wont be hung

Hi,

if it goes for JDBC Connection you should check the min pool size configured and set this to 0 as this avoids having stale connections in pool.

Regards,
Holger

well we already set the jdbc into 0 min pool size

Hello Holger,
honestly we allready set the min pool into zero
but its still happening

here some setting for the jdbc:

is there any suggestion?

best regards

Can you share the config of the JDBC service? Is there a timeout configured for it?

hello reamon
this is our jdbc timeout setting
but, it didnt work when there is a hung thread
i thougt that this setting will kill the hung thread,
but I realized that this timeout were works for killing the connection not the thread

cmiiw

@Tegar_ASK ,
Have you confirmed with your Oracle DBA’s that they are not seeing any database deadlocks on the tables that the Integration Server’s JDBC Adapter operates on?
Just curious as you have 300 connections.
Regards
~Jerry

well, they have confirmed that, at the time threads were hung
there was no deadlock on Oracle Session
it was really confusing what make the threads hung
and I wonder what should we do to these hung threads without entrupting the jvm
I ever read the same issue and there is a java script that could kill the thread immadiatelly but it will impact the jvm and gonna make another issue I guest

Best Regards

Killing threads does not quality as a solution and I’m strongly against using a script or any “fix” to kill threads - it will just be a band-aid fix and cause a ton more problems across all domains.

Do get back to us after checking the points I mentioned above - I suspect locked and waiting threads.

KM

Is this same adapter connection being used for reading and writing to the same tables which the DBA’s have seen locks on?

yup, the exact same username access and adapter sir

Hi @Tegar_ASK ,
This might be a case of one adapter connection trying to auto-commit a record on a record or page while another connection is trying to read data from the same block.
You might have to think of the folloing approach

• Keep your existing services which perform insert’s, updates, deletes using the same adapter connection.
  -Create a new adapter connection to the same database only for read purposes with the settings of transaction isolation (provided it suits your requirements) such as being able to ready uncommitted data.

This is just a though -btw changing the reading adapter service which does the read is not as hard as it seems you could refer to pub.art.service:setAdapterServiceNodeConnection which can do this task without your team having to rebuild the code.

Regards
~jerry

thankyou Kasi for your advice
we won’t do killing the thread,
we gonna setting up our jdbc as you mention above
I do have the same suspect as yours…

Best Regards

Adding detail to the settings you shared:

Block timeout is the amount of time a thread will wait for a pool connection before giving up and throwing an error. This has nothing to do when an individual adapter service waits for a response. This applies only to threads waiting to get a connection from the pool.

Expire timeout is the amount of time a connection in the pool is allowed to remain idle. If not used within the expire timeout, the connection will be closed (should be set to just short of what the DB idle connection timeout is).

There is a timeout on the adapter service itself. The adapter connection settings you list are for the connection session, not the execution of a statement. The fun part is when it is created it defaults to -1 which means “wait forever.” Should never do that on any adapter calls (DB or other) nor on any HTTP calls.

Not sure if that is the case here but is something to check.