Data Protection Software

 View Only
  • 1.  Restoring 16 Million Files (TSM + ESS3000) improve performance

    Posted Fri April 30, 2021 05:20 AM
    Hello,

    We are doing backup / restore tests with a new storage (ESS3000) and we are trying to restore a directory tree with 16 million files (5 Tb)
    The bakcup (mmbackup) does not go bad, but the restore takes too long, just listing the files (ANS1247I Waiting for files from the server) takes between 90-120 minutes.
    So we decided to restore a smaller subdirectory of only 2.6 million, the file listing takes 15 to 20 minutes and the actual restoration takes 3 to 4 hours.

    TSM SERVER 7.9 (AIX)   TSM client 8.11 (ESS3000 EMS linuxPLE)

    MAXNUMMP=16
    Txnbytelimit 2097152
    resourceutilization 10

    any subjection to improve performance ??

    Thanks and Regards ???

    ------------------------------
    Luis Tejeda Alcolea
    ------------------------------


  • 2.  RE: Restoring 16 Million Files (TSM + ESS3000) improve performance

    Posted Fri April 30, 2021 11:11 AM
    Hola Louis, 
        Though I may not have the technical skill nor experience to help you - I did ask few colleagues at IBM. One of the advices was to check out the following technote: How to initiate an optimized restore operation for a large number of objects stored by IBM Spectrum Protect on tape
        I understand you are not using the tape but it may give you a hint at what to look at.

        If that does not help, I would suggest you contact IBM support and we will make sure your problem gets assistance and is resolved. 

    thank you and good luck !

    ------------------------------
    Zoran Hrustic
    ------------------------------



  • 3.  RE: Restoring 16 Million Files (TSM + ESS3000) improve performance

    IBM Champion
    Posted Mon May 03, 2021 02:13 AM
    change diskbuffsize with different values in dsm.sys and try again.
    default is 256, first try with decreased values like 32, 64 and then try to increase like 512 and compare the results.

    ------------------------------
    NEZIH BOYACIOGLU
    ------------------------------



  • 4.  RE: Restoring 16 Million Files (TSM + ESS3000) improve performance

    Posted Mon May 03, 2021 09:39 AM

    I think the solution is SOBAR that Spectrum Scale has.

    https://www.ibm.com/docs/en/spectrum-scale/5.0.2?topic=scale-introduction-out-backup-restore-sobar

    Thanks
    Christian



    ------------------------------
    Christian Svensson
    Storage Consultant
    Crisie Nordic AB
    ------------------------------



  • 5.  RE: Restoring 16 Million Files (TSM + ESS3000) improve performance

    Posted Tue May 18, 2021 02:59 AM
    Edited by Tom Van Daele Tue May 18, 2021 02:59 AM

    I would indeed recommend a combination of what you doing now with SOBAR or snapshotting.
    The incremental backup process is good for individual file restores but 16M files would be quite the job for any Spectrum Protect server.

    In regards to supermassive (file)systems it is always best to try and incorporate some form of snapshotting, in combination with INC/DIFF. It wil save you a lot of time!


    For the restores now try and play with multiple sessions, TCP settings (tcpbuffsize tcpnodelay), transaction parameters (diskbuffsize, txnbytelimit) to optimize your restore process, but be advised 16M files will always take a substantial ammount of time.  



    ------------------------------
    Tom Van Daele
    ------------------------------