Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hi Florian,

Did you check the performance counters provided by the Listeners?
https://www.ibm.com/support/knowledgecenter/SSAE9L_4.0.1/com.ibm.content.collector.doc/monitoring/r_afu_pf_counters.htm 

FileNet and ICC, including these Listener performance and capacity metrics, can be monitored realtime by IBM ECM System Monitor: IBM ECM System Monitor Overview - maximize the service quality of your ECM applications  

I hope that helps a bit.

Regards,
Roland

------------------------------
Roland Merkt
Sr Manager EIM
CENIT
------------------------------

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hi Florian,

Have you checked to see if virus scanning software is running on the server?  If so, it is worth white listing the working directories for ICC so that the virus scanner isn't processing every file processed.  If CPU is running at 100% then that would say it needs more CPU (assuming that no other processes are consuming the CPU like virus scanning).

Regards
Phil
IBM Automation Expert Labs

------------------------------
PHILIP RIMMINGTON
------------------------------

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hi... this is a basic recommendation but is very important too. A couple year ago I had this situation in a customer's production environment. After disabling the AV checking for the work folders of ICC, the performance of the server came back to normal and it's working until now with no additional tuning or resources.


------------------------------
Sergio Salinas
------------------------------

Original Message:
Sent: Thu February 25, 2021 03:45 AM
From: PHILIP RIMMINGTON
Subject: IBM Content Collector : how to tune it efficiently ?

Hi Florian,

Have you checked to see if virus scanning software is running on the server?  If so, it is worth white listing the working directories for ICC so that the virus scanner isn't processing every file processed.  If CPU is running at 100% then that would say it needs more CPU (assuming that no other processes are consuming the CPU like virus scanning).

Regards
Phil
IBM Automation Expert Labs

------------------------------
PHILIP RIMMINGTON

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hello all,

Very sorry for my delayed answer due to several emergencies to handle :(

So we are still exploring the way to improve ICC performance.
Concerning antivirus, our input is on a NAS and KO, OK output folders are on this NAS too. And it seems that NAS is bound to antivirus too. So file are scanned when push to input folder but on output folders too. So, as soon we resolve our RACI problem, we can deal with it  lol. Thank you to Philip and Sergio for this suggestion. Note that antivirus on the server seems not to be involved as no CPU consumption for it.

Thank you Roland for your suggestion concerning counters, we will use it. Concerning ECM System Monitor, it could be difficult to sell but it worths a try.

I opened a ticket on support concerning monitoring via Dynatrace, but for the moment nothing interesting.

I will keep you in touch on this topic as soon as possible.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Original Message:
Sent: Fri February 26, 2021 08:47 AM
From: Sergio Salinas
Subject: IBM Content Collector : how to tune it efficiently ?

Hi... this is a basic recommendation but is very important too. A couple year ago I had this situation in a customer's production environment. After disabling the AV checking for the work folders of ICC, the performance of the server came back to normal and it's working until now with no additional tuning or resources.


------------------------------
Sergio Salinas

Original Message:
Sent: Thu February 25, 2021 03:45 AM
From: PHILIP RIMMINGTON
Subject: IBM Content Collector : how to tune it efficiently ?

Hi Florian,

Have you checked to see if virus scanning software is running on the server?  If so, it is worth white listing the working directories for ICC so that the virus scanner isn't processing every file processed.  If CPU is running at 100% then that would say it needs more CPU (assuming that no other processes are consuming the CPU like virus scanning).

Regards
Phil
IBM Automation Expert Labs

------------------------------
PHILIP RIMMINGTON

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

There are a couple of ICC performance and sizing related documents on Seismic. Please check them out and let me know in case of any remaining questions.

------------------------------
Jörg Stolzenberg
------------------------------

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hello Jörg,

I looked in Seismic.
I checked "ICC Sizing + Best Practices Wiki" file but if I understand, we can just expect 20 files/second for 1 server with a size of 5-200Kb. And no information concerning 100% CPU consumption...
Concerning Red papers almost 10 years old (!), they are interesting but very focused on email archiving.

We need more information on why (or leads) ICC services can consume 100% of CPU. And how to reduce it. Meanwhile, we need abacus on file archiving and what we can expect on our architecture.

Regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Original Message:
Sent: Wed February 24, 2021 10:59 AM
From: Jörg Stolzenberg
Subject: IBM Content Collector : how to tune it efficiently ?

There are a couple of ICC performance and sizing related documents on Seismic. Please check them out and let me know in case of any remaining questions.

------------------------------
Jörg Stolzenberg

Original Message:
Sent: Tue February 23, 2021 10:55 AM
From: Florian KIEBEL
Subject: IBM Content Collector : how to tune it efficiently ?

Hello all,

I wonder if this forum is the best spot to post on IBM Content Collector, but it worths the try.

So my question is actually 2 questions :

1. What are the best settings to tune in order to have an efficient rate of document injection between ICC and FileNet P8 5.5.x ?
   
   Currently we have a virtual server with 2 cores configured with 16 threads and 128 queue size and document injection rate is about 10,8 document / sec with 100% CPU consumption for a batch of 30000 documents. This is good but, I think we can do better with the same resources.
   1. Typically, Task Route Engine service consumes about 45% and FileSystem Connector almost 40%, unlike P8 Connector consuming is consuming only 7%. Why Task Route Engine and FileSystem Connector take so much CPU ?
   2. Memory (RAM) is always a 1/3 of available memory during the injection round.
   3. Note that our ICC routes take documents from folder in NAS share and then move documents in OK/KO folder on the same NAS share.
   4. Does anyone have any relevant experience on optimizing ICC architecture and document injection rate please ?
      
      
2. We are trying to monitor deeply ICC with Dynatrace tool which is an APM. But currently Dynatrace is not recognising processes and so we cannot drill and introspect ICC processes like Task Routing Engine, FileSystem Connector or P8 Connector. Has anyone tried to introspect ICC with performance tooling ? If yes, any advices on this topic ?

I will open a ticket on IBM support on this but I wondered if anyone got this kind of behaviour before.

Thanks for your help.

Best regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------