I am not familiar with the new API so I cannot comment on that.
Since you mentioned sweeps, I am assuming uploads and not downloads. Each object storage will have different tuning options. If a customer is doing mortage docs, the bulk of retrievals will happen in the first 30 days, for another customer it might be 3 days. After that window, the docs are less likely to be retrieved so you might choose cool storage to save money and after a year change it to cold storage. Cloud storage is cost efficient for archive storage. It may also be the only option if the organization storage policy does not allow docs with PII even if encrypted.
Cloud storage also offers simplified redundancy. You can fail over VM's or containers from one DC to another easily if the storage is loosely coupled and replicated. However, if you don't have cloud redundancy/failover, then you need some sort of backing store to be able to retry uploads when they fail. I saw two storage outages last year; when Microsoft updated their root cert and again when they added a new gateway. It only impacted retrieval of older docs but you never want to see any outage from an operations perspective.
Latency is environment specific. The situation where it was running full tilt on weekends was low volume, but multi-tenant, with multiple customer VM's per host and the storage was in another cloud vendor's data center. I never understood the reason behind that but I assume it was either cost or naively specced. I suggest that if you're running in the cloud on AWS, use S3, if you're running in Azure, use Azure if possible. You can always run a trace route for comparison. If you're on-premises you really shouldn't have any problems with either unless your pipe is undersized or the connection is poor.
If you're communicating across continents the latency could be elsewhere. I have twice seen customers outsource validation to India or South America but refuse to replicate LDAP so the authorization occurred on-premises over a slow VPN. Even if the storage was replicateted across data centers, the Dashboard metrics showed the lastency and timeouts were occurring because DNS lookups and LDAP authorizations were taking at least 2 minutes and could intermittently timeout under load.
Since you mentioned sweeps, if the total number of rows in the docVersion table isn't too large, sweeps will work and they run very fast. Even if the swep is 23x faster, a sweep cannot target a specific storage area. It would be nice if the content migration sweep had an option to specify a source storage area (hint). Instead, it looks at all rows and processes them by object_id and applies a filter, e.g.,
StorageArea=Object('{F0A47194-0000-CF1D-8122-41E6201DF3D4}')
and VersionStatus=1
and DateCreated < NOW() - TimeSpan(1, 'Days')
I would create/run a sample sweep to get an idea of the time and then make your own decision. My alternate approach worked for me because the sweep takes 3 days to complete and I wanted to move the docs to Azure sooner than that. Using a query, I could target just the staging storage area with a query (where date_created > X and storage_area_id = Y). Be sure to validate your composite index gets chosen by the optimizer. That sweep runs in under an hour and the docs are moved the next day.
It's really impossible to offer specific advice without knowing the specifics so the above is fairly broad but hopefully gives you some ideas.
------------------------------
Stephen Weckesser
------------------------------
Original Message:
Sent: Mon February 24, 2025 10:08 AM
From: Kiran Aithagani
Subject: FileNet S3 Retrieval Latency
I saw that in one of the readme's but I think it come int play when you use the API but for S3 it is a set configuration and not sure if we have can control to choose single thread vs multi threaded approach, unless 5.6 s3 configuration is different/has that option than 5.5.10 CPE
------------------------------
Kiran Aithagani
Original Message:
Sent: Mon February 24, 2025 09:46 AM
From: Chuck Hauble
Subject: FileNet S3 Retrieval Latency
Kiran,
In the v5.6 API there is a new API class that will allow you to do a multithreaded download from the CPE - com.filenet.apiimpl.util.mtstream.MtInputStream
The download by default is single threaded.
Maybe it could help with the problem. But I see that you are using ICN instead of a custom application so it may require an enhancement request to the ICN team.
------------------------------
Chuck Hauble
Minneapolis
Original Message:
Sent: Fri February 21, 2025 03:35 PM
From: Kiran Aithagani
Subject: FileNet S3 Retrieval Latency
Hi Stephen, very interesting thoughts, thank you for sharing the ideas, I am following up with IBM with a case, as we were anticipating some latency with S3 storage but it is beyond the acceptance and we would like to know the bottleneck and look into finetuning options to make it better, we do have some thoughts on directing the content to the different storage area based on the size, using sweep or some sort of the subscription etc.
Thanks
Kiran Aithagani
------------------------------
Kiran Aithagani
Original Message:
Sent: Fri February 21, 2025 01:18 PM
From: Stephen Weckesser
Subject: FileNet S3 Retrieval Latency
Cloud storage will always have latency issues. I suggest to keep new docs in a local 'staging area' for a period of time and then move them to cloud storage later. Docs are more likely to be retrieved when they are new. After a few days (it will vary by use case) the frequency will decline. You can use a migration policy sweep and filter on the storage area if your containers are broken down by date. This also works well if long term storage is write only (e.g. has a retention policy). It also allows for a window of time allow deletes during a validation period. A sweep will work but if you have a lot of documents the sweep may take a few days to complete before it starts over. I use a custom app that searches for docs by date in the staging area and gets kicked off by a scheduler. Since the separate staging area is small (HA and backed up), it only has to examine a few thousand docs instead of every row in the doc version table and I pass the name of the config file so I can run multiple instances that take different parameters. The doc move method does not physically move the document, it just queues it for the server to move. This approach keeps the dba's happier because sweeps block and they no longer get constant alerts in the middle of the night. With this, I control the number of rows and time each run is allowed. If you store directly to the cloud, the system may get backed up without anyone noticing. Files may sit in a temp upload area on the server until the weekend comes along. When nobody is expecting it, the retry kicks in and the system will run hot and can get overwhelmed resulting in memory constraints one day, followed by high cpu load the next as it tries to run garbage collection. This also allows for uploads to continue if the cloud storage does down. (or the vendor changes the certs without notifying you... Again Azure?)
------------------------------
Stephen Weckesser
Original Message:
Sent: Thu December 12, 2024 04:24 PM
From: Olaf Erasmus
Subject: FileNet S3 Retrieval Latency
I'm sorry it did not help.
Thank you for the detailed feedback though, valuable for anyone considering to store large objects on S3.
------------------------------
Olaf Erasmus
Original Message:
Sent: Thu December 12, 2024 11:46 AM
From: Kiran Aithagani
Subject: FileNet S3 Retrieval Latency
Olaf, thank you for your suggestion, Yes I did experiment with different Max Inline Content Sizes on CPE first and tried downloading the file from CPE, irrespective of what my Max Inline Content set to, CPE is always downloading the document in the chunks of 1MB
But when I set FileNet.Content.GetBlockSizeKB on CN, and try to view/download the documents then I get bigger chunks from S3, matching up to the FileNet.Content.GetBlockSizeKB value I provided, which is good, however when I get bigger chunks from S3 , it is taking longer to stich those pieces together and stream to Content Navigator.
Below are not exact numbers but I tried changing the FileNet.Content.GetBlockSizeKB ranging from 1MB to 9MB (setting Max Inline Content Size to 10MB on CPE), over all one way or the other the latency is adding up to the same amount, may be 1-2 seconds less or more, but not significant change with any of the combinations we tried.
- when I do default 1 MB chunks , it is doing 30 round trips and takes about 25 seconds to retrieve and +5 seconds to stich them together and stream the doc to CN.
- when I set the FileNet.Content.GetBlockSizeKB to 8 MB, its doing 4 round trips, and finishing up the download in 15 seconds + 15 seconds to stich them and stream the doc to CN
------------------------------
Kiran Aithagani
Original Message:
Sent: Mon December 09, 2024 07:26 AM
From: Olaf Erasmus
Subject: FileNet S3 Retrieval Latency
Have you tried setting the FileNet.Content.GetBlockSizeKB parameter on the ICN server to higher value?
The multiple round trips to S3 might exponentially increase the overall latency.
https://www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=engine-improving-content-uploads-downloads
------------------------------
Olaf Erasmus
Original Message:
Sent: Fri December 06, 2024 04:02 PM
From: Kiran Aithagani
Subject: FileNet S3 Retrieval Latency
Hello,
We have an on-prem FileNet Content Manager setup, version 5.5.10 and CN 3.0.13, both CN and CPE are on a Linux virtual server. For content storage we have NetApp storage now, to leverage AWS S3 storage we also configured advanced storage as an alternative option.
We ran some tests to se the performance of retrieval , from both on-prem NetApp and Cloud S3 storage, up until 4-5 MB sizes the documents load in Content Navigator Viewer almost the same time, but as the documents get bigger and bigger the S3 retrieval show significant latency, like for a 15 MB , 12-15 second and for 30 MB. 20 -25 seconds.
Wondering anybody tried this in their environments and ran into similar issue and have any suggestions to try.
Thanks
Kiran Aithagani
------------------------------
Kiran Aithagani
------------------------------