Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.
Hi All,
Can anyone help how we can kill long-running services that are continuously running on Windows?
We tried the below 3 options one by one but didn’t help and still services are running continuously.
IS Version - 9.9
Hi Yogeshwar,
any specifics about this service, why it is continously running? Why do you think, that it is continously running?
Are there any fixes applied to the IS? Any messages in the server.log related to this service?
Try to capture some thread dumps from the JVM process and check if there are some blocked threads and what they are waiting for. You will find these thread dumps in the wrapper.log under profiles/IS_default/logs.
Regards, Holger
I triggered a process that will pull 14000+ records and update them in DB, so this process got stuck and later we tried to cancel the process and saw a huge count on service usage for the main and child services. To stop this, we performed all the above steps but didn’t work.
We didn’t apply any latest IS fixes and didn’t see any errors on the server.log file.
Below are some of the Server Thread Dump logs:
“TriggerTask:35:Test.svc.trigger:subScribeLoadT&D” Id=195 in RUNNABLE
“HTTP Handler 10.72.145.38” Id=194 in WAITING
“Thread-103” Id=193 in TIMED_WAITING
“Statistics Thread-SoftwareAG.IS.WMN-1” Id=191 in TIMED_WAITING
“SoftwareAG.IS.WMN” Id=190 in WAITING
“Statistics Thread-KFPay-1” Id=188 in TIMED_WAITING
“KFPay” Id=187 in WAITING
“Statistics Thread-SoftwareAG.IS.ART-1” Id=185 in TIMED_WAITING
“SoftwareAG.IS.ART” Id=184 in WAITING
“Statistics Thread-SoftwareAG.IS.Core-1” Id=182 in TIMED_WAITING
“SoftwareAG.IS.Core” Id=181 in WAITING
“Statistics Thread-SoftwareAG.IS.PE-1” Id=179 in TIMED_WAITING
“SoftwareAG.IS.PE” Id=177 in WAITING
“Statistics Thread-SoftwareAG.IS.Services-1” Id=175 in TIMED_WAITING
“SoftwareAG.IS.Services” Id=174 in WAITING
“pool-3-thread-1” Id=168 in TIMED_WAITING
“Thread-89” Id=167 in TIMED_WAITING
“Thread-88” Id=166 in TIMED_WAITING
“Thread-87” Id=165 in TIMED_WAITING
“Thread-86” Id=164 in TIMED_WAITING
“Thread-85” Id=163 in TIMED_WAITING
“Thread-84” Id=162 in TIMED_WAITING
“Thread-83” Id=161 in TIMED_WAITING
“Thread-82” Id=160 in TIMED_WAITING
“Thread-81” Id=159 in TIMED_WAITING
“Thread-80” Id=158 in TIMED_WAITING
“Thread-79” Id=157 in TIMED_WAITING
“Thread-78” Id=156 in TIMED_WAITING
“Thread-77” Id=155 in TIMED_WAITING
“Timer-11” Id=153 in TIMED_WAITING
“Timer-10” Id=148 in TIMED_WAITING
“RMI RenewClean-[10.70.112.7:8075]” Id=138 in TIMED_WAITING
“RMI Scheduler(0)” Id=137 in TIMED_WAITING
“GC Daemon” Id=135 in TIMED_WAITING
“RMI Reaper” Id=134 in WAITING
“RMI TCP Accept-8075” Id=133 in RUNNABLE (running in native)
“http-bio-8074-AsyncTimeout” Id=132 in TIMED_WAITING
“http-bio-8074-Acceptor-0” Id=131 in RUNNABLE (running in native)
“DeviceManager” Id=128 in WAITING
“http-bio-8072-AsyncTimeout” Id=127 in TIMED_WAITING
“http-bio-8072-Acceptor-0” Id=126 in RUNNABLE (running in native)
“derby.rawStoreDaemon” Id=122 in TIMED_WAITING
“Timer-7” Id=121 in TIMED_WAITING
“fileinstall-C:\SoftwareAG\common\nerv\bundles” Id=120 in TIMED_WAITING
“Timer-6” Id=119 in TIMED_WAITING
“Thread-50” Id=112 in TIMED_WAITING
“Thread-49” Id=111 in TIMED_WAITING
“Thread-48” Id=110 in TIMED_WAITING
“UUIDMonitor-Thread-47” Id=109 in TIMED_WAITING
“UUIDManagerDaemon-Thread-46” Id=108 in TIMED_WAITING
“TxnData-Deadlock-Manager” Id=107 in TIMED_WAITING
“Camel (com.softwareag.eda.nerv.default.consume.context-6) thread #2 - ManagementLoadTask” Id=106 in TIMED_WAITING
“com.softwareag.eda.nerv.caches” Id=102 in WAITING
“Camel (com.softwareag.eda.nerv.default.emit.context-4) thread #1 - ManagementLoadTask” Id=101 in TIMED_WAITING
“FSData Free Cache Handler” Id=100 in WAITING
“Thread-40” Id=99 in WAITING
“UM Transport high accuracy tick timer” Id=88 in TIMED_WAITING
“Monitor Thread” Id=87 in TIMED_WAITING
“Thread-28” Id=85 in WAITING
“ISM Log Thread” Id=84 in WAITING
“DailyFileAppenderTimer” Id=83 in TIMED_WAITING
“Camel (camel-1) thread #0 - ManagementLoadTask” Id=82 in TIMED_WAITING
“Thread-25” Id=81 in TIMED_WAITING
“Thread-24” Id=80 in TIMED_WAITING
“Cron Daemon” Id=79 in TIMED_WAITING
“Cron Daemon Pool2” Id=78 in WAITING
“Cron Daemon Pool1” Id=76 in WAITING
“Timer-5” Id=77 in WAITING
“Timer-4” Id=75 in WAITING
“Thread-21” Id=74 in RUNNABLE (running in native)
“RunWatchdogThread” Id=73 in WAITING
“ContainerBackgroundProcessor[StandardEngine[Catalina]]” Id=72 in TIMED_WAITING
“Thread-19” Id=69 in WAITING
“webMethods Integration Server” Id=68 in WAITING
“Thread-18” Id=67 in TIMED_WAITING
“Integration Server Launcher” Id=65 in WAITING
“Thread-9” Id=54 in TIMED_WAITING
“Statistics Thread-SSOAssertionsCacheManager-1” Id=51 in TIMED_WAITING
“SSOAssertionsCacheManager” Id=50 in WAITING
“Configuration watchdog 1” Id=46 in TIMED_WAITING
“Spring DM Context Creation Timer” Id=45 in WAITING
“Spring DM Context Creation Timer” Id=44 in WAITING
“[WireAdmin] - Event Dispatcher” Id=43 in WAITING
“Provisioning Event Dispatcher” Id=42 in WAITING
“Worker-0” Id=41 in TIMED_WAITING
“[ThreadPool Manager] - Idle Thread” Id=40 in WAITING
“[Timer] - Main Queue Handler” Id=39 in TIMED_WAITING
“Worker-JM” Id=37 in WAITING
“Timer-1” Id=35 in WAITING
“Bundle File Closer” Id=34 in WAITING
“Log4J mirrors cache cleaner” Id=32 in WAITING
“Timer-0” Id=31 in TIMED_WAITING
“Framework Event Dispatcher” Id=23 in WAITING
“Start Level Event Dispatcher” Id=22 in WAITING
“Framework Active Thread” Id=20 in TIMED_WAITING
“DestroyJavaVM” Id=16 in RUNNABLE
“Wrapper-Connection” Id=15 in RUNNABLE
“Wrapper-Control-Event-Monitor” Id=13 in TIMED_WAITING
“Attach Listener” Id=6 in RUNNABLE
“Signal Dispatcher” Id=5 in RUNNABLE
“Finalizer” Id=3 in WAITING
“Reference Handler” Id=2 in WAITING
Normally cancel/kill the service from IS->Server->Statistics page or restart the IS should work. If it is for a development purpose I suggest to restart your Windows.
Hi @Yogeshwar.Kabra Is this happening every time you trigger the service which pulls 14000+ records and update them in DB?
Try to do
As you are requesting large set of transaction for DB, it might be getting locked in DB, try to check in DB logs why its getting locked and try to fix it.
@Kanojia_Vivek01 @Sourabh_Harkawat2 @codrelphi - Thanks for your reply,
@codrelphi - IS restarted twice and didn’t help much, Windows box is not restarted. Even tried IS->Server->Statistics and killed the process but keep increasing the process again and again.
@Sourabh_Harkawat2 - This is happening only on the Dev environment, we have Pub-Sub service so all records are published but on the subscribe side it’s running continuously, even trigger properties are the same as PROD.
@Kanojia_Vivek01 - On DB checked but nothing shows as long-running queries/procs.
I can see the below error on the server log when I ran the process again.
[SoftwareAG][SQLServer JDBC Driver]Login has timed out.
java.sql.SQLException: [SoftwareAG][SQLServer JDBC Driver]Login has timed out.
at com.wm.dd.jdbc.sqlserverbase.ddcw.b(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.ddcw.a(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.ddcv.b(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.ddcv.a(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.ddak.a(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.BaseConnection$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at com.wm.dd.jdbc.sqlserverbase.BaseConnection.b(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.BaseConnection.a(Unknown Source)
at com.wm.dd.jdbc.sqlserverbase.BaseDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.wm.app.jdbc.JDBCConnection.(JDBCConnection.java:51)
at com.wm.app.jdbc.JDBCConnectionPool.getAvailConnection(JDBCConnectionPool.java:409)
at com.wm.app.jdbc.JDBCConnectionPool.getConnection(JDBCConnectionPool.java:223)
at com.wm.app.jdbc.JDBCConnectionPool.getConnection(JDBCConnectionPool.java:203)
at com.wm.app.b2b.server.scheduler.ScheduleDB.getConnection(ScheduleDB.java:1090)
at com.wm.app.b2b.server.scheduler.ScheduleDB.runNextTask(ScheduleDB.java:1454)
at com.wm.app.b2b.server.scheduler.ScheduleManager.run(ScheduleManager.java:201)
at com.wm.util.pool.PooledThread.run(PooledThread.java:127)
at java.lang.Thread.run(Thread.java:745)
Failed to remove the expired entries from the document history table. See server log for details. Caused by: java.sql.SQLException: [SoftwareAG][SQLServer JDBC Driver]Login has timed out.
Trigger Properties: IS UM Connection used
Check if you can disable the adapter connection. There is no reason for a service to automatically start after IS restart under normal circumstances, unless you add it to scheduled tasks or trigger it manually or from a publishable document/queue. There might be a firewall blockage that is preventing the adapter connection to connect to db. Are you triggering the process from a document or some other way? If that’s the case, try disabling the trigger and then purging the documents. If the process isn’t triggered from UM or a publishable document, there is no need to check UM connections.
check if there are any duplicate triggers as well.
You want to kill the Integration server service or Winnows services? For windows service kill from Task manager or recycle the box.