Content Management and Capture

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hello Duke Lam,

Thanks for your response. I've already gone through the section "System scalability, availability, backup, and recovery" from the Datacap Redbook, as well as the book "IBM Datacap 9.0.1 Performance Scalability, Tuning and Best Practices." However, I will revisit the "System scalability, availability, backup, and recovery" section from the Redbook again.

Regarding the bottleneck, we have observed that the most affected processes are:

1. wTM requests (stuck at the IIS Worker processor, as attached) ex "queue/createBatch,Rules/Excute, and loadeuserprofiler",
2. Batch creation,
3. Export task Profiler Rulesets (convertPDFToTIFF, export to FileNet, DeleteOriginalImages).

Could you clarify what you mean by "splitting"? Are you referring to:

1. Splitting the batch folder?
2. Reducing the number of pages per batch (though this isn't valid for us)?
3. Using Datacap Batch Splitting?

Your guidance on this matter would be greatly appreciated.

Best regards,

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

I will certainly do my best to answer. 

wTM request causing IIS worker process to stop, hang, terminate? 
Is this using "out of the box " datacap navigator conducting batch scanning or are you using a custom REST api call?  Some thing is cause IIS to terminate or hang?   Isolate if that is the case.  Is it terminating IIS or is it just taking too much system CPU resources used?  I believed RedBook has a comment about 100 to 150 user for each IIS server.   In wTM log you can see user logon and follow the session ID from batch entry to batch leave. 

For splitting batch folder. 
I was going off on your comments "We are considering splitting the batch folder to allow each web server to work on a separate folder" 
If you are using RR server for batch creation, Each application should only have one batch creation in the RR server thread. 
If you are using a app to do file ingestion and considering lower the scan source image directory.   I suppose it can create two folder.  Folder A and Folder B and places your image in two folder.  You can have a workflow or a rule set to look for image in Folder A only and another to scan for image in folder B. 
Sort of lower the time to read the entire source image folder if there are alot.  But you can use Mvscan as it's smart to scan the source folder and keep track of what is in the folder and using a lock and unlock to remember what it is pulling in for ingesting.  

Can't comment of batch splitting yet as I don't think that is your issue.  Unless the batch are huge with lots of pages. 

Hello Duke Lam,

Thanks for your response. I've already gone through the section "System scalability, availability, backup, and recovery" from the Datacap Redbook, as well as the book "IBM Datacap 9.0.1 Performance Scalability, Tuning and Best Practices." However, I will revisit the "System scalability, availability, backup, and recovery" section from the Redbook again.

Regarding the bottleneck, we have observed that the most affected processes are:

1. wTM requests (stuck at the IIS Worker processor, as attached) ex "queue/createBatch,Rules/Excute, and loadeuserprofiler",
2. Batch creation,
3. Export task Profiler Rulesets (convertPDFToTIFF, export to FileNet, DeleteOriginalImages).

Could you clarify what you mean by "splitting"? Are you referring to:

1. Splitting the batch folder?
2. Reducing the number of pages per batch (though this isn't valid for us)?
3. Using Datacap Batch Splitting?

Your guidance on this matter would be greatly appreciated.

Best regards,

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hello Duke Lam,

Thank you very much; I truly appreciate your time and effort.

1. WTM request causing IIS worker process to stop, hang, terminate?  
   - The wTM request, it's causing the IIS worker process to hang and not serve any new requests, rather than stopping or terminating. This results in the user login screen continuously loading until the error "repository not available" is raised. Interestingly, system CPU usage is not significantly impacted, averaging between 1% and 10%.
   
   
2. Scanning or are you using a custom REST api call?
   We are using Datacap Navigator only-no custom REST API calls are involved.

For server configuration, we have two servers, each with 30 cores and 64GB RAM, handling about 5,000 concurrent users. Based on the recommendation of handling 100 to 150 users per IIS server, what would you suggest as the optimal configuration for this setup?

Regarding splitting the batch creation process, since it is triggered through Datacap Navigator (wTM), and we have two web services (wTMs) and two application servers (TMs), I believe we can configure each wTMs server to create batches on a separate server.

Looking forward to your insights.

I will certainly do my best to answer. 

wTM request causing IIS worker process to stop, hang, terminate? 
Is this using "out of the box " datacap navigator conducting batch scanning or are you using a custom REST api call?  Some thing is cause IIS to terminate or hang?   Isolate if that is the case.  Is it terminating IIS or is it just taking too much system CPU resources used?  I believed RedBook has a comment about 100 to 150 user for each IIS server.   In wTM log you can see user logon and follow the session ID from batch entry to batch leave. 

For splitting batch folder. 
I was going off on your comments "We are considering splitting the batch folder to allow each web server to work on a separate folder" 
If you are using RR server for batch creation, Each application should only have one batch creation in the RR server thread. 
If you are using a app to do file ingestion and considering lower the scan source image directory.   I suppose it can create two folder.  Folder A and Folder B and places your image in two folder.  You can have a workflow or a rule set to look for image in Folder A only and another to scan for image in folder B. 
Sort of lower the time to read the entire source image folder if there are alot.  But you can use Mvscan as it's smart to scan the source folder and keep track of what is in the folder and using a lock and unlock to remember what it is pulling in for ingesting.  

Can't comment of batch splitting yet as I don't think that is your issue.  Unless the batch are huge with lots of pages. 

Hello Duke Lam,

Thanks for your response. I've already gone through the section "System scalability, availability, backup, and recovery" from the Datacap Redbook, as well as the book "IBM Datacap 9.0.1 Performance Scalability, Tuning and Best Practices." However, I will revisit the "System scalability, availability, backup, and recovery" section from the Redbook again.

Regarding the bottleneck, we have observed that the most affected processes are:

1. wTM requests (stuck at the IIS Worker processor, as attached) ex "queue/createBatch,Rules/Excute, and loadeuserprofiler",
2. Batch creation,
3. Export task Profiler Rulesets (convertPDFToTIFF, export to FileNet, DeleteOriginalImages).

Could you clarify what you mean by "splitting"? Are you referring to:

1. Splitting the batch folder?
2. Reducing the number of pages per batch (though this isn't valid for us)?
3. Using Datacap Batch Splitting?

Your guidance on this matter would be greatly appreciated.

Best regards,

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hello Ahmed, 

You have two tms servers and two wtms.  You are using native datacap navigator.  no custom REST.  Performing scanning task and seeing IIS or w3wp.exe appears to hang or not responding.   When other user try to logon, you see the generic repository not loading.  I am wondering if this is all cause by your scan task.
If you are to take scan task out of the equation, do you see this issue with wTM and IIS having issue?  I am think that you are doing a huge scan volume.  These are physical scanner or just vscan as in pulling image from a drive?  So, the bottleneck appears to be cause by scan task and potentially upload delay.  What you can do is set upload to asycn or set you max upload size.

Max upload see #1 

1. https://www.ibm.com/docs/en/datacap/9.1.8?topic=support-uploading-large-size-document-in-datacap-navigator

Else you should this first to see if IIS or w2wp.exe is hanging. 

Option #2 Aycn upload

1. Log in to Datacap Navigator.
2. Go to Datacap administration view, and click Workflows.
3. Double-click a workflow from the list, and click the Jobs tab.
4. Double-click any job, and then click the Tasks tab.
5. Double-click any task, go to the Start Batch Panel section, and select the Async Upload check box.

I think it's either the scan task upload taking time and causing IIS issue.  Setting it to Async force the upload task will run the background freeing up IIS resources.   That or the size of your batch are huge and Option #1 may help in that url.  

Try to isolate scan task is the culprit causing IIS to hang.

Hello Duke Lam,

Thank you very much; I truly appreciate your time and effort.

1. WTM request causing IIS worker process to stop, hang, terminate?  
   - The wTM request, it's causing the IIS worker process to hang and not serve any new requests, rather than stopping or terminating. This results in the user login screen continuously loading until the error "repository not available" is raised. Interestingly, system CPU usage is not significantly impacted, averaging between 1% and 10%.
   
   
2. Scanning or are you using a custom REST api call?
   We are using Datacap Navigator only-no custom REST API calls are involved.

For server configuration, we have two servers, each with 30 cores and 64GB RAM, handling about 5,000 concurrent users. Based on the recommendation of handling 100 to 150 users per IIS server, what would you suggest as the optimal configuration for this setup?

Regarding splitting the batch creation process, since it is triggered through Datacap Navigator (wTM), and we have two web services (wTMs) and two application servers (TMs), I believe we can configure each wTMs server to create batches on a separate server.

Looking forward to your insights.

I will certainly do my best to answer. 

wTM request causing IIS worker process to stop, hang, terminate? 
Is this using "out of the box " datacap navigator conducting batch scanning or are you using a custom REST api call?  Some thing is cause IIS to terminate or hang?   Isolate if that is the case.  Is it terminating IIS or is it just taking too much system CPU resources used?  I believed RedBook has a comment about 100 to 150 user for each IIS server.   In wTM log you can see user logon and follow the session ID from batch entry to batch leave. 

For splitting batch folder. 
I was going off on your comments "We are considering splitting the batch folder to allow each web server to work on a separate folder" 
If you are using RR server for batch creation, Each application should only have one batch creation in the RR server thread. 
If you are using a app to do file ingestion and considering lower the scan source image directory.   I suppose it can create two folder.  Folder A and Folder B and places your image in two folder.  You can have a workflow or a rule set to look for image in Folder A only and another to scan for image in folder B. 
Sort of lower the time to read the entire source image folder if there are alot.  But you can use Mvscan as it's smart to scan the source folder and keep track of what is in the folder and using a lock and unlock to remember what it is pulling in for ingesting.  

Can't comment of batch splitting yet as I don't think that is your issue.  Unless the batch are huge with lots of pages. 

Hello Duke Lam,

Thanks for your response. I've already gone through the section "System scalability, availability, backup, and recovery" from the Datacap Redbook, as well as the book "IBM Datacap 9.0.1 Performance Scalability, Tuning and Best Practices." However, I will revisit the "System scalability, availability, backup, and recovery" section from the Redbook again.

Regarding the bottleneck, we have observed that the most affected processes are:

1. wTM requests (stuck at the IIS Worker processor, as attached) ex "queue/createBatch,Rules/Excute, and loadeuserprofiler",
2. Batch creation,
3. Export task Profiler Rulesets (convertPDFToTIFF, export to FileNet, DeleteOriginalImages).

Could you clarify what you mean by "splitting"? Are you referring to:

1. Splitting the batch folder?
2. Reducing the number of pages per batch (though this isn't valid for us)?
3. Using Datacap Batch Splitting?

Your guidance on this matter would be greatly appreciated.

Best regards,

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hi Duke Lam,

Thank you so much for your effort.

Yes, the issue arises when each server performs heavy tasks on the storage, whether it's the wTMs servers (handling scanning tasks) or the RuleRunner servers (handling the conversion from TIFF to PDF and DeleteImages tasks). We are currently working to enhance the Storage performance to handle the heavy load.

I also want to ask you:

1. What do you mean by isolating the scan task? How can this be achieved?
2. We have also observed a lot of the following requests:
   • /ServicewTm.svc/Session/LogonExitloadUserProfile=true: Would setting LoadUserProfile=false in IIS improve performance?
   • /ServicewTm.svc/Rules/Execute: Do you have any recommendations for improving the execution of these requests?

I really appreciate your help and support!

Best regards,

Hello Ahmed, 

You have two tms servers and two wtms.  You are using native datacap navigator.  no custom REST.  Performing scanning task and seeing IIS or w3wp.exe appears to hang or not responding.   When other user try to logon, you see the generic repository not loading.  I am wondering if this is all cause by your scan task.
If you are to take scan task out of the equation, do you see this issue with wTM and IIS having issue?  I am think that you are doing a huge scan volume.  These are physical scanner or just vscan as in pulling image from a drive?  So, the bottleneck appears to be cause by scan task and potentially upload delay.  What you can do is set upload to asycn or set you max upload size.

Max upload see #1 

1. https://www.ibm.com/docs/en/datacap/9.1.8?topic=support-uploading-large-size-document-in-datacap-navigator

Else you should this first to see if IIS or w2wp.exe is hanging. 

Option #2 Aycn upload

1. Log in to Datacap Navigator.
2. Go to Datacap administration view, and click Workflows.
3. Double-click a workflow from the list, and click the Jobs tab.
4. Double-click any job, and then click the Tasks tab.
5. Double-click any task, go to the Start Batch Panel section, and select the Async Upload check box.

I think it's either the scan task upload taking time and causing IIS issue.  Setting it to Async force the upload task will run the background freeing up IIS resources.   That or the size of your batch are huge and Option #1 may help in that url.  

Try to isolate scan task is the culprit causing IIS to hang.

Hello Duke Lam,

Thank you very much; I truly appreciate your time and effort.

1. WTM request causing IIS worker process to stop, hang, terminate?  
   - The wTM request, it's causing the IIS worker process to hang and not serve any new requests, rather than stopping or terminating. This results in the user login screen continuously loading until the error "repository not available" is raised. Interestingly, system CPU usage is not significantly impacted, averaging between 1% and 10%.
   
   
2. Scanning or are you using a custom REST api call?
   We are using Datacap Navigator only-no custom REST API calls are involved.

For server configuration, we have two servers, each with 30 cores and 64GB RAM, handling about 5,000 concurrent users. Based on the recommendation of handling 100 to 150 users per IIS server, what would you suggest as the optimal configuration for this setup?

Regarding splitting the batch creation process, since it is triggered through Datacap Navigator (wTM), and we have two web services (wTMs) and two application servers (TMs), I believe we can configure each wTMs server to create batches on a separate server.

Looking forward to your insights.

I will certainly do my best to answer. 

wTM request causing IIS worker process to stop, hang, terminate? 
Is this using "out of the box " datacap navigator conducting batch scanning or are you using a custom REST api call?  Some thing is cause IIS to terminate or hang?   Isolate if that is the case.  Is it terminating IIS or is it just taking too much system CPU resources used?  I believed RedBook has a comment about 100 to 150 user for each IIS server.   In wTM log you can see user logon and follow the session ID from batch entry to batch leave. 

For splitting batch folder. 
I was going off on your comments "We are considering splitting the batch folder to allow each web server to work on a separate folder" 
If you are using RR server for batch creation, Each application should only have one batch creation in the RR server thread. 
If you are using a app to do file ingestion and considering lower the scan source image directory.   I suppose it can create two folder.  Folder A and Folder B and places your image in two folder.  You can have a workflow or a rule set to look for image in Folder A only and another to scan for image in folder B. 
Sort of lower the time to read the entire source image folder if there are alot.  But you can use Mvscan as it's smart to scan the source folder and keep track of what is in the folder and using a lock and unlock to remember what it is pulling in for ingesting.  

Can't comment of batch splitting yet as I don't think that is your issue.  Unless the batch are huge with lots of pages. 

Hello Duke Lam,

Thanks for your response. I've already gone through the section "System scalability, availability, backup, and recovery" from the Datacap Redbook, as well as the book "IBM Datacap 9.0.1 Performance Scalability, Tuning and Best Practices." However, I will revisit the "System scalability, availability, backup, and recovery" section from the Redbook again.

Regarding the bottleneck, we have observed that the most affected processes are:

1. wTM requests (stuck at the IIS Worker processor, as attached) ex "queue/createBatch,Rules/Excute, and loadeuserprofiler",
2. Batch creation,
3. Export task Profiler Rulesets (convertPDFToTIFF, export to FileNet, DeleteOriginalImages).

Could you clarify what you mean by "splitting"? Are you referring to:

1. Splitting the batch folder?
2. Reducing the number of pages per batch (though this isn't valid for us)?
3. Using Datacap Batch Splitting?

Your guidance on this matter would be greatly appreciated.

Best regards,

Refer to Datacap Redbook on scaling the systems.  Where is the bottleneck?  Ingesting/scan, pageID, profiler, etc?

If it file ingestion where the bottleneck is happening on then splitting the batch would make sense as each scan task is looking at different image source location to increase file I/O.  

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

You can decrease the RRS log level on Rulerunner Manager so that Rulerunner threads won't write log messages to the shared disk much. 

Also, you should turn off log flush option if it is on.

If your custom actions are writing to the shared disk, you can consider decreasing them or buffer them before writing.

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

Hi Dsakai,

Thanks for your response.

We've already disabled the log flush option and reduced the RRS log level to minimize log writing and size. Additionally, we don't have any custom actions configured.

Let me know if there are any other recommendations you can share.

Best regards,

You can decrease the RRS log level on Rulerunner Manager so that Rulerunner threads won't write log messages to the shared disk much. 

Also, you should turn off log flush option if it is on.

If your custom actions are writing to the shared disk, you can consider decreasing them or buffer them before writing.

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?

You say Shared Disk is the bottle neck.

Then, I think you can reduce the number of Tasks in your application workflow so that Rulerunner threads won't write to the disk much.

They will process data in memory. Also, I've never tried this before, but if possible you can let applications use the local disk on each Rulerunner Server

to do temporary processing. This will reduce the I/O on the shared disk. 

Hi Dsakai,

Thanks for your response.

We've already disabled the log flush option and reduced the RRS log level to minimize log writing and size. Additionally, we don't have any custom actions configured.

Let me know if there are any other recommendations you can share.

Best regards,

You can decrease the RRS log level on Rulerunner Manager so that Rulerunner threads won't write log messages to the shared disk much. 

Also, you should turn off log flush option if it is on.

If your custom actions are writing to the shared disk, you can consider decreasing them or buffer them before writing.

Hi,

We are experiencing a significant number of batches in a pending status, and the wTM requests are getting stuck at the IIS, as shown in the attached logs. After investigating, we discovered that the Datacap system is facing poor performance when accessing the shared batch folder. Our system is currently handling around 40k batches, with over 1 million pages, all created within about 4 hours.

Our system architecture includes:

• 4 RuleRunner servers
• 2 TMs
• 2 wTMs
  Each server is equipped with 30 cores and 64GB of RAM.

Has anyone else faced similar issues with the storage setup?
We are considering splitting the batch folder to allow each web server to work on a separate folder, though it would still be on the same storage.

Are there any recommendations to improve the system performance?