Content Management and Capture

 View Only
Expand all | Collapse all

Migrating Large Document Number between Object Store Using FDM

  • 1.  Migrating Large Document Number between Object Store Using FDM

    Posted Mon January 16, 2023 12:45 PM

    Hi,

    I'm planning to migrate a large number of documents (approx. 600k in total) from the existing Object Store to newly created Object Store on Ms.SQL Server.

    The default Storage Areas set on current/existing Object Store is just using Default Database Storage. So the database size for this OS is growing bigger.
    So we need to create a new object store that using both default database storage and filestorage area.

    I hope I can achieve that and migrate those documents through FileNet Deployment Manager. Is there any limitation regarding of this case?

    Or any insight on other alternatives that can be done to achieve the proper solution.

    Thanks,
    Dhandy 



    ------------------------------
    Dhandy Yusuf Sahyadi
    ------------------------------


  • 2.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Tue January 17, 2023 04:39 AM
    Hi Dandhy,
    if your goal is to move the documents from one storage area to another, you can use the FileNet sweep subsystem.
    Take a look to the following documentation pages to see if sweeps can help.
    Handling bulk processing with sweeps: https://www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=objects-handling-bulk-processing-sweeps
    Moving content with sweeps: https://www.ibm.com/docs/en/filenet-p8-platform/5.5.x?topic=content-creating-job-sweep

    Massimiliano

    ------------------------------
    Massimiliano Carra
    Advisory Business Automation Technical Specialist
    IBM Technology, Italy
    ------------------------------



  • 3.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Thu February 02, 2023 09:43 PM
    Thank you for the input and reference, Massimiliano

    ------------------------------
    Dhandy Yusuf Sahyadi
    ------------------------------



  • 4.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Tue January 17, 2023 08:47 AM
    Hi Dhandy,

    You don't need to create a new objectstore to add the File Storage Area. you can just add it to the existing objectstore and then as Massimiliano said use the Sweep framework to move the documents. 

    You can use FDM to move them from one ObjectStore to another, but that process will be 
    tedious.

    ------------------------------
    Chuck Hauble
    Minneapolis MN
    ------------------------------



  • 5.  RE: Migrating Large Document Number between Object Store Using FDM
    Best Answer

    Posted Tue January 17, 2023 09:26 AM
      |   view attached
    Dhandy

    Assuming you are on a reasonably current version of FileNet, you don't actually need to create a new object store to resolve your issue. Instead, use the sweep capability in ACCE to move the existing content to a different storage area.

    The sweep will run in the background, so your object store stays usable throughout the process.

    At a high-level, here's the suggested procedure

    1. Add a new storage area & policy to the object store that uses the storage you want to use going forward.
    2. Update your document classes to use the new storage policy....that way all new ingest will go to the new storage areas rather than database storage
    3. Use the bulk move option in the Sweep to move your existing content from database storage to the new storage areas
    4. Once the move is complete, reorg your database to reclaim space

    As with all things, please try this out in a non-production environment first.

    More information on performing bulk actions with sweep is available here: Handling bulk processing with sweeps. I'm also attaching a presentation on the capability.

    ------------------------------
    RUTH Hildebrand-Lund
    rhildebr@us.ibm.com
    ------------------------------

    Attachment(s)

    pdf
    Chicago - Sweep Framework.pdf   1.33 MB 1 version


  • 6.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Fri February 03, 2023 04:36 AM
    I was involved in the past to tests and develop tools for such large documents commitment to P8 or FileNet Image services in the past...and I found it very interesting.. I forward your documentation to my team I was working Maybe they contact you who knows...

    ------------------------------
    dorothea vulcan
    dorotheavulcandorotheavulcan@yahoo.comdorotheavulcandorotheavulcan@yahoo.com
    ------------------------------



  • 7.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Tue February 07, 2023 10:47 PM

    Thank RUTH Hildebrand-Lund for giving a clear explanation regarding to my case.

    Do you have any technical paper/publication regarding pro-cons of using Database Storage Area vs File Storage Area?

    And also how to perform backup & recovery procedure using File Storage Area?

    Thank you,
    Dhandy Yusuf Sahyadi



    ------------------------------
    Dhandy Yusuf Sahyadi
    ------------------------------



  • 8.  RE: Migrating Large Document Number between Object Store Using FDM

    IBM Champion
    Posted Wed February 08, 2023 06:05 AM
    Advantage of db storage area:

    1. One backup backs it up all, at restore no sync between any storage and db is not necessary (and not possible)
    2. I have seen performance tests that show slight performance gains with small objects

    Disadvantage:

    The database usually gets VERY large bringing with it all sorts of operational problems.

    I have yet to see a customer that uses the db storage area for the majority of documents... sometimes they store a limited set (e.g. templates,forms) there

    Advantage File Storage

    1. Wide range of options to choose from
    2. Simple to configure/manage

    Disadvantage:

    Can run out of sync with the database.
    At restore time you should really know what you are doing.
    Security might be a problem in critical environments as you can access the share with an appropriate user and delete or encrypt it with some ransom

    My personal preference for storage (our milage may vary) is object based storage in aligned mode like ICOS, ECS or HCP.

    Kind regards,

    Gerold

    Kind regards,

    ------------------------------
    Gerold Krommer
    ------------------------------



  • 9.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Wed February 08, 2023 01:12 PM
      |   view attached
    Dhandy

    You can find a good description of the different types of storage areas we support in the P8 documentation...take a look at this link: Storage area types

    Depending on the type of storage area you choose, there are more choices regarding the physical storage that you can use...take a look at this technote for those details: https://www.ibm.com/support/pages/node/6592835
    (There is also a reference to this technote on the hardware tab in the software product compatibility report for FileNet. (You can generate the compatibility reports on this website: IBM Software Product Compatibility Reports

    Depending on the type of storage you select, your backup process will be different. If you choose to use file storage, then you also have a choice of how you write to the storage areas, and this in turn, will also affect your backup process. I've attached a technote that provides more information on strategies for using file storage areas. While the technote is old, the information is still very applicable to the newer releases of FileNet.

    Hope this all makes sense

    ------------------------------
    RUTH Hildebrand-Lund
    rhildebr@us.ibm.com
    ------------------------------

    Attachment(s)



  • 10.  RE: Migrating Large Document Number between Object Store Using FDM

    Posted Tue January 17, 2023 10:25 AM
    Hi Dhandy, if you only need to move the content you can use ACCE and a sweep job. However as your question specifically states moving content between object stores and FDM I have not had any luck moving >  750 documents reliably. You will need to build a tool to copy - migrate these documents or a packaged custom solution as I offer that moves data between ostores and P8 systems, preserves the ID, VSID, Creator, modifier, versions, etc. Check out my ECM workstation software for ~1 million document migrations per day or if you need a larger product with database backend for ongoing migrations with asynch importers that's also available a bit more on the cost side.

    ------------------------------
    Jay Bowen
    ------------------------------