IBM Storage Defender

IBM Storage Defender

Early threat detection and secure data recovery

 View Only
Expand all | Collapse all

SAP RMAN backup is taking too much time to complete

  • 1.  SAP RMAN backup is taking too much time to complete

    Posted Sun March 10, 2019 07:24 AM
    Hi All,

    I am looking for your guidance on how to handle SAP backup which is working through RMAN, it is taking almost 18-20 hours to complete the backup, database size is 3.5 TB.

    I am using RMAN channel 2, I am looking for a option to set below option to speed up the backup process.

    Multiplexing- For me i guess tsm doesn't support.
    Parallel Session- I can see through BRtools i can use this but on which i am using RMAN_util it is not working

    I am not able to see option where to update this while using RMAN_UTIL
    Parallel sessions :
    Multiplexed files :
    Buffer size :

    if any other parameter which i am missing please advise that as well, backup is happening through STA, that is why i am not convince it is taking that much time.

    SID.sap few details.
    # number of members in RMAN savesets [ 1 | 2 | 3 | 4 | tsp | all ]
    # default: 1
    saveset_members = 1

    # additional parameters for RMAN
    # rman_channels and rman_filesperset are only used when rman_util
    # rman_channels defines the number of parallel sbt channel allocations
    # rman_filesperset = 0 means:
    # one file per saveset - for non-incremental backups
    # all files in one saveset - for incremental backups
    # the others have the same meaning as for native RMAN
    #rman_channels = 1
    rman_channels = 2
    # rman_filesperset = 0
    # rman_kbytes = 0
    # rman_readrate = 0
    # rman_maxopenfiles = 0
    # rman_setsize = 0
    # additional parameters for RMAN version 8.1
    # the parameters have the same meaning as for native RMAN
    # rman_diskratio = 0
    # rman_pool = 0
    # rman_duplex = 0 | 1 | 2 | 3 | 4
    # rman_proxy = no | yes | only
    rman_parms="SBT_LIBRARY=/oracle/PB1/112_64/lib/libobk.sl,ENV=(XINT_PROFILE=/oracle/PB1/112_64/dbs/initPB1.utl,PROLE_PORT=57323,&BR_INFO)"
    # remote copy-out command (backup_dev_type = pipe)
    # $-character is replaced by current device address
    # no default
    copy_out_cmd = "dd ibs=8k obs=64k of=$"

    ------------------------------
    Prem Singh

    ------------------------------


  • 2.  RE: SAP RMAN backup is taking too much time to complete

    Posted Thu March 14, 2019 03:02 AM
    For each RMAN channel a session to the server or STA is opened. If you have more drives you could try increasing the number of channels. Don't forget to increase MAX_SESSIONS.

    But I agree that 3.5 T in 18 hours is rather slow. Even for a single drive. Probably you are limited by read performance. Would recommend to follow the tuning instructions in the Oracle doc -> https://docs.oracle.com/en/database/oracle/oracle-database/12.2/bradv/tuning-rman-performance.html#GUID-33D81037-2310-4021-8039-1A2C579AAA5A


  • 3.  RE: SAP RMAN backup is taking too much time to complete

    Posted Thu March 14, 2019 03:57 AM
    Thanks Thomas,

    Thank you for the help but session and other configuration is a part of util file and when I am using rman_util it doesn't go to util file for any check I have 7 drives for this backup and STA , do your have any option to share me any sample where I have to change sessions in .sap file.

    Thanks 
    Prem 





  • 4.  RE: SAP RMAN backup is taking too much time to complete

    Posted Thu March 14, 2019 08:31 AM
    You can specify parameter rman_channels in .sap file.

    But again: average throughput of 50 MB/s will not keep two drives in streaming mode. You better consider to tune the disk reads.

    ------------------------------
    Thomas Prause
    ------------------------------



  • 5.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 18, 2019 01:14 AM
    I have tried to assign 2 and 4 both  one by one but no improvement as such.

    Thanks 
    Prem





  • 6.  RE: SAP RMAN backup is taking too much time to complete

    Posted Fri March 15, 2019 10:33 AM
    If your situation hasn't changed from some of your previous questions you are using tape drives not disk drives for storage. This will limit your backup speed to the write speed of your tape drives.

    For tape drive backups there are two MAX_SESSIONS related to Spectrum Protect, not rman. The first is the maximum total number of sessions for all backups.

    1. Check to see what the maximum sessions that are allowed to be used overall is: "query option maxsessions"
    I believe you want this one set to at least the number of drives you have. See "h maxsessions". For disk backup this can be much higher.

    2, Now check to see how many sessions are allowed for the SAS node you are backing up.
    "select node_name, max_mp_allowed from nodes where node_name='Your SAS node name'"

    The Max_mp_allowed is the maximum number of tape drives that can be used. Make sure it is set to at least the number of rman sessions you want to use. Each rman session will open another connection to the Spectrum Protect server. If the Max_mp_allowed is set to 1 which is the default the Spectrum Protect server will only use 1 tape drive and will only allow one connection from the SAS server.

    To set the MAX_SESSIONS if needed.

    UPDATE NODE 'Your SAS node name' maxnummp=2

    My understanding is that if you change to using something other than tape the MAX_MP_ALLOWED (aka maxnummp) is no longer relevant.

    ------------------------------
    Lynn Hall
    ITS 4
    OCIO State of Iowa
    Des Moines IA
    515 669 4325
    ------------------------------



  • 7.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 18, 2019 01:20 AM
    Hi Lyn,

    Thank you for the help and detail, yes this is tape drive Lto 5 , the settings I have checked earlier as well but no link why because when just change the option into .sap file from rman to use util file it just change the situation tremendously and backup completed within 4 hours only. That confused me what is wrong when I use rman one more thing I notice after one set of backup through RMAN it wait for more than 10 mnts to send next set that could be one of the cause but I don't find anywhere where to change this settings.



    Thanks
    Prem 





  • 8.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 18, 2019 12:06 PM
    If the backup with BR*Tools device type 'util_file' using 4 sessions does complete within 4 hours, then you proved that the performance is not limited by the number of sessions/drives. Adding even more sessions is not going to help.

    Seems RMAN is spending too much time reading the data. Tuning RMAN is beyond my knowledge. But would recommend to check if any validation options are turned on. Don't ask me any details. I'm just guessing that RMAN does 'waste' time while doing checks of the data read. Sorry that I can not be of more use.

    ------------------------------
    Thomas Prause
    ------------------------------



  • 9.  RE: SAP RMAN backup is taking too much time to complete

    Posted Tue March 19, 2019 07:14 AM
    Thanks Thomas,

    I do agree there is somewhere in the settings validation or checking is happening, which is the real cause, I am working on it with other db teams I will post if anything fruitful come out from any changes.

    Thanks 
    Prem 





  • 10.  RE: SAP RMAN backup is taking too much time to complete

    Posted Fri March 15, 2019 11:25 AM
    ​Hi Prem,

    Questions for you:
    When you observe the job running, how many sessions do you see?  Are there tape mounts for each of the sessions?
    What is your maxnummp setting on the node?
    How many drives are zoned to the storage agent?
    Is your vtl a physical tape library of a virtual tape libary?  If virtual, which system are you using?
    Are you using physical or virtual hba's?
    I am assuming your rman script says to open 7 sessions, right?
    Do you use the same node name for filesystem backups as you do for database backups?



    ------------------------------
    Brenda Collins
    Sr. Principal Storage Admin
    Medtronic
    Moundsview MN
    ------------------------------



  • 11.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 18, 2019 01:26 AM
    Hi Brenda,

    Thanks for the question here is my output.


    Questions for you:
    When you observe the job running, how many sessions do you see?  Are there tape mounts for each of the sessions?- 4 sessions always, I believe no only 2 tapes at a time.
    What is your maxnummp setting on the node?12
    How many drives are zoned to the storage agent?7
    Is your vtl a physical tape library of a virtual tape libary?  If virtual, which system are you using? it's Physical tape I am using lto5 and IBM 3500 library
    Are you using physical or virtual hba's?Physical
    I am assuming your rman script says to open 7 sessions, right?rman I use only 2 or 4 channels in .sap file but while I use util file option it has 4 sessions settings.
    Do you use the same node name for filesystem backups as you do for database backups?No it is tdp node and file system backup is from another node.

    Pls let me know if you need more information to help me further.


    Thanks 
    Prem 
    Mobile 





  • 12.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 18, 2019 11:48 AM

    Hi Prem

     

    You must have a role that takes care of both databases and Spectrum Protect.  I don't so I cannot really talk to the database side other than what  I see.  We use DB2 and they do have a comment in their scripts telling it to use the number of sessions.  Example:  db2 backup db PDL ONLINE use tsm open 5 sessions DEDUP_DEVICE with 30 buffers buffer 4097 INCLUDE LOGS 

     

    I don't see that listed in your statement: ' I use only 2 or 4 channels in .sap file but while I use util file option it has 4 sessions settings'

     

    Where are you identifying that you want the job to use 7 sessions?  I haven't worked with rman for some time now so I am unsure of the command the use. You do have to define somewhere how many sessions to use for that job.

     

    Is the storage agent running at the time you start your backups?  You should be able to see one or more sessions showing for the storage agent.  It is possible that even though it shows running, it is actually hung for some reason.  You could go onto the client and restart the storage agent just in case prior to testing your next session.

     

    Brenda

     

    This message has been marked as Medtronic Controlled

     






  • 13.  RE: SAP RMAN backup is taking too much time to complete

    Posted Tue March 19, 2019 07:16 AM
    Hi Brenda,

    I will post the settings today when I will be in office then everyone will understand what I am pointing.

    Thanks 
    Prem 





  • 14.  RE: SAP RMAN backup is taking too much time to complete

    Posted Sun March 24, 2019 07:40 PM
    Hi Brenda,

    Please find the snip which i am pointing .





    Now all of sudden I am seeing my backup window increased in both the cases. I am attaching the zoned drives as well something is going very weird.

     
    It has so many drives assigned that doesn't mean it will use all but it will use only those which is available for  that duration.

    ------------------------------
    Prem Singh

    ------------------------------



  • 15.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 25, 2019 03:58 PM

    Hi Prem,

     

    I still don't see where you are telling the job to use seven drives.  Maybe a RMAN  expert can comment on where that is set in a RMAN script.  It looks like you are defining the same 10 drives to two different Storage Agents in your picture.  The issue with this is:

    1. The same 10 drives are assigned to: 
      1. FRMRS133_sta
      2. FRMRS132_sta
      3. TSM Server itself

    As a result, you could have a scenario where both systems are running db backups at the same time as well as admin jobs on the TSM server.  If one client wants 7 drives and actually takes them, that only leaves 3 left for everything else.  If you are trying to run both rman backups in the same time frame, they will just get what is available regardless of how many you tell it to use.  It is possible the one job will fail completely if it can't get enough drives. 

     

    You will also need to ensure there are no admin process taking away the use of your drives.  Hopefully, you have your jobs scheduled to avoid the conflicts.  If your job is to defined to use 7 drives and it doesn't mount 7, it has to be a command in your script or the fact that there are not enough drives available.  The command 'q mount' will show you how many drives are in use in total.   If backups are running on frmrs133_sta, you can run the command 'frmrs133_sta: q se '  and it will show you how many mounts are for that specific storage agent'.

     

    I have also seen backups where it may start out using a certain number of drives and then periodically drop down to fewer drives and possible start up more sessions again later.  That happened due to the size of the tables within the database and it was DB2 related.    I am not sure I am helping you this time around due to not working with RMAN for a while.

     

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com





  • 16.  RE: SAP RMAN backup is taking too much time to complete

    Posted Mon March 25, 2019 05:18 PM
    Hi Brenda,

    I have shared the snap where RMAN_Channels was 2 as a test i change it to 4-6 but no luck. per channel is =1 drive since all the drives are zoned it can occupy whatever the free available on that duration.  

    1. FRMRS133_sta - These 2 are same cluster node
    2. FRMRS132_sta  These 2 are same cluster node 
    Just to share the example i have pasted that, Admin jobs most of the time is not running on that duration I have 20 + SAP Backup almost all are 3+ TB and i have only 9 drives in all , why i am confused because where STA is not available on that system it is taking almost the same time to complete the backup although when I do FRMRS132_sta:q sess i can sess 4 session i f i have given 4 channels but that is not at all convincing me , Do I have any other option to check further for me for sure it is not using STA or STA is not working i have killed and restarted STA several times but so far nothing new. 

    --
    Regards
    Prem