Sometimes the searchTask API call fails with a Read Timeout Exception generated on the IS logs. Going through the MWS logs, i do see the call is logged and is completed.

2016-08-09 22:00:05 EDT (task:INFO)  - Query All Tasks Task Search Query [max:200] searchTerms: {#{currentTask.taskData.myTask.employeeID} = 12345; name = My Task; status = active}
2016-08-09 22:01:08 EDT (Framework:INFO) [WS:360061] - WebService Request: "command/searchtasks_ws" completed in 62721

This, to me means that MWS is able to fetch the task (if any) from the database, and return the same back to the service call. On the IS however, i see the timeout exception:

electric.util.WrappedException: java.net.SocketTimeoutException: Read timed out
at com.webmethods.caf.wsclient.proxy.impl.WSClientDynamicProxy.getCompatibleException(WSClientDynamicProxy.java:496)
at com.webmethods.caf.wsclient.proxy.impl.WSClientDynamicProxy.invoke(WSClientDynamicProxy.java:257)
at $Proxy32.searchTask(Unknown Source)
at pub.task.taskclient.searchTasks(taskclient.java:194)
at sun.reflect.GeneratedMethodAccessor18415.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:443)
at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:643)
at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:41)
at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:243)
at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:51)
at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:276)
at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:30)
at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:363)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:547)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:386)
at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:234)
at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:194)
at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:584)
at com.wm.lang.flow.FlowState.step(FlowState.java:444)
at com.wm.lang.flow.FlowState.invoke(FlowState.java:409)
at com.wm.ap...<truncated>

I tried increasing the timeout parameter on the WmTaskClient package to 2 mins (from the default 1 min), but the issue still persists, (it does not occur all the time, but it does occur). Is there some other configuration parameter i should be looking at ?

This is on v8.2.

Thanks,
Janardan.

Hi Janardan,

can you try searchTaskFields instead of searchTask?

This will reduce the payload to be transferred between IS and MWS when selecting the correct fields.

Regards,
Holger

HI Janardan,

In addition to the service, as you are running query on task data. To speed up the search it will be better to INDEX the field.

Hi Team,

I know this is very old thread but we are facing this issue on 9.12 environment.

Once this searchTasks API starts saying Read time out, rest all API’s are failing and it is hung untill MWS is restarted.

Once we restart all of a sudden this happens in day or two.

This is happening in Prodution where it ran fine for 2 weeks after go live but started failing intermittently.

I have increased the timeout to 2 minutes in WmTaskClient and also increased the wm_mon_services timeout to two minutes.

Still this keeps happening.

What is meant by Index here.?

Hi Team,

Please find below logs for the issue

BPM Server logs:

2020-03-23 07:49:25 UTC [BPM.0102.0002E] Exception: com.wm.app.b2b.server.ISRuntimeException
2020-03-23 07:49:25 UTC [BPM.0102.2001E] c25542c0-cf68-4e80-9cf2-d6b3bb9e8a67:1: exception invoking wm.task.taskclient:controlTasks service: [98.9998] Exception --> 0 electric.util.WrappedException: java.net.SocketTimeoutException: Read timed out 

MWS Server logs during this time:

2020-03-23 07:49:08 UTC (notifications:WARN) [MWS Event Queue (Inbound pool-9-thread-1] - PortletNotificationDeliverer: portlet null is not IPortalEvent.IListener
2020-03-23 07:50:03 UTC (Framework:INFO) [qtp1691742369-5383] [RID:11672] - Request [12bos3dm02y6v6hp3wnud1xsj:nai] http://MWS HOST:8585/meta/default/wm_xt_fabricfolder/0000006802 (POST) 
2020-03-23

Once we restart the MWS server, it is working fine.

I have increased the wm_mon_services WS timeout to 2 minutes. WmTaskClient timeout to 2 minutes. But still the issue persist.

any suggestions are really helpful to resolve this issue. We aren’t able to re-produce in lower environments and it is happening in Production.

Thanks.

There is a KB article which points to same error as in your post. See if this helps:

If MWS has the JVM option -Dtask.prt.audit.enabled=true when a task which belongs to a business process is created, MWS will audit records on the ProcessAudit database.

ProcessAudit database is unknown to MWS, it will try to obtain the database info with the following order:

1- If db.xml contains a datasource named ProcessAudit it will be used.
2- If 1 is not found, check if same schema for MWS contains ProcessAudit data, by executing the SQL: SELECT INSTANCEID FROM PRA_PROCESS WHERE INSTANCEID IS NULL
3- If 1 and 2 has no results MWS will execute service WmMonitor/wm.task.taskclient:getProcessAuditJDBCConfig from the IS defined under MWS > Applications > Administration > My webMethods > System Settings > TaskEngine

During the migration db.xml will be updated to same URL as MWS, if ProcessAudit is defined you will need to double check if the JDBC URL and username used are correct for the new ProcessAudit database, otherwise change the value manually with proper information.

@Mahesh_K_Sreenivas

All these URLs are updated after migration to point to the right Process Audit database.

The issue was occurring without any pattern. All of a sudden it happens one fine day.

By restarting the MWS, the issue gets resolved by itself. So currently team is doing maintenance activity auto-restarting MWS.

Hi Firoz,

hope you are doing well.

pub.task.taskclient:searchTasks service returns data based on the search criteria. If the search criteria relatively returns smaller set of data, it may be quick but if it returns huge number of records, this service will take more time. Also, this API examines all business data in all the tasks in the inbox. Increasing timeout would be a temporary solution as it may affect when returned dataset is huge.

pub.task.taskclient:searchTasksIndexed is an API that searches only on indexed fields. In your task data, you will specify which fields to be indexed (again, not all but the fields that really need to be indexed based on use-case). This API will work much better as it uses DB index.

This goes back to implementation phase but if that is an option, you should consider.

To start with, I suggest you look at MWS performance (one of the built-in hidden tool)

Documentation: https://techcommunity.softwareag.com/ecosystem/documentation/webmethods/wmsuites/wmsuite9-12/My_webMethods/9-12_Diagnosing_My_webMethods_Server.pdf

Section: Using the Performance Statistics Portlet

By enabling this portlet, you can see many parameters one of which is the Task API’s time taken, DB query time taken etc., This would give you an indication to find root cause

Regards
Senthil

@Senthilkumar_G.

Thanks for asking. I’m fine. Hope you are doing well. Long time

Appreciate your response and found definitely useful. In future during upgrade program, I will take it up to change the API to index one for better performance as suggested.

Also I verified few options in MWS diagnosis but will go through more on the same and see if there is an option.

I feel somewhere the memory/thread is not getting released and causing this random occurrences like once in 4 days. So I will forward the support team all this analysis.

Regards,
Firoz Nalband.

Hi,

You could try calling the TaskClient API directly in the IS with the same parameters to check if the time it takes to return the answer is as long as the one you are experiencing in the MWS.

Also, raising the log level for the WmMonitor sections might help you measuring the response time.

I have seen that sometimes WmMonitor does return rather quickly but the amount of the data retrieved has the MWS taking a long time to render.

Monitoring the MWS internals (for instance, enabling its JMX port and using the Java Mission Control or VisualJVM tools) might help you understand what is going on.

Probably increasing the MWS’s JVM max reserved memory might help, but it is better to check what the above tools tell you.

Best regards,

Hi Gerardo,

I will explore on the suggestions and options provided by you and see what we can find.

Appreciate your response.

Whats the wsclient-socketTimeout? You can try and increase it until adopting the better performing API. You can check what is the setting at global level under: MWS > Folders > Administrative Folders > Administration Dashboard > Configuration > CAF Application Runtime Configuration → Configure Global Defaults
If you suspect a memory issue heapdump is the best option to validate the suspicion.
Have a nice day!

Hi Bgmeat,

We tried increasing that but it didn’t help much either.