Content Management and Capture

 View Only
  • 1.  FileNet S3 Retrieval Latency

    Posted Fri December 06, 2024 04:02 PM

    Hello,

    We have an on-prem FileNet Content Manager setup, version 5.5.10 and CN 3.0.13, both CN and CPE are on a Linux virtual server.  For content storage we have NetApp storage now, to leverage AWS S3 storage we also configured advanced storage as an alternative option.

    We ran some tests to se the performance of retrieval , from both on-prem NetApp and Cloud S3 storage, up until 4-5 MB sizes the documents load in Content Navigator Viewer almost the same time, but as the documents get bigger and bigger the S3 retrieval show significant latency, like for a 15 MB , 12-15 second and for 30 MB. 20 -25 seconds.

    Wondering anybody tried this in their environments and ran into similar issue and have any suggestions to try.

    Thanks

    Kiran Aithagani



    ------------------------------
    Kiran Aithagani
    ------------------------------


  • 2.  RE: FileNet S3 Retrieval Latency

    Posted Mon December 09, 2024 09:36 AM

    Hi Kiran,
    Unfortunately I do not have good advice for you, however, I am curious if you have attempted to utilize NFS against your on-prem NetApp storage to see if it has the same behavior?



    ------------------------------
    Matt Passini
    Manager - Software Engineering
    Acuity Insurance
    ------------------------------



  • 3.  RE: FileNet S3 Retrieval Latency

    Posted Thu December 12, 2024 09:19 AM

    Have you tried setting the FileNet.Content.GetBlockSizeKB parameter on the ICN server to higher value?

    The multiple round trips to S3 might exponentially increase the overall latency. 

    https://www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=engine-improving-content-uploads-downloads



    ------------------------------
    Olaf Erasmus
    ------------------------------



  • 4.  RE: FileNet S3 Retrieval Latency

    Posted Thu December 12, 2024 11:47 AM

    Olaf, thank you for your suggestion, Yes I did experiment with different Max Inline Content Sizes on CPE first and tried downloading the file from CPE, irrespective of what my Max Inline Content set to, CPE is always downloading the document in the chunks of 1MB

    But when I set FileNet.Content.GetBlockSizeKB on CN, and try to view/download the documents then I get bigger chunks from S3, matching up to the FileNet.Content.GetBlockSizeKB value I provided, which is good, however when I get bigger chunks from S3 , it is taking longer to stich those pieces together and stream to Content Navigator.

    Below are not exact numbers but I tried changing the FileNet.Content.GetBlockSizeKB ranging from 1MB to 9MB (setting Max Inline Content Size to 10MB on CPE), over all one way or the other the latency is adding up to the same amount, may be 1-2 seconds less or more, but not significant change with any of the combinations we tried.

    1. when I do default 1 MB chunks , it is doing 30 round trips and takes about 25 seconds to retrieve and +5 seconds to stich them together and stream the doc to CN.
    2. when I  set the FileNet.Content.GetBlockSizeKB to 8 MB, its doing 4 round trips, and finishing up the download in 15 seconds + 15 seconds to stich them and stream the doc to CN


    ------------------------------
    Kiran Aithagani
    ------------------------------



  • 5.  RE: FileNet S3 Retrieval Latency

    Posted Thu December 12, 2024 04:25 PM

    I'm sorry it did not help. 

    Thank you for the detailed feedback though, valuable for anyone considering to store large objects on S3. 



    ------------------------------
    Olaf Erasmus
    ------------------------------