Data Protection Software

Expand all | Collapse all

SAP Backup getting failed for multiple Server on muliple instance

  • 1.  SAP Backup getting failed for multiple Server on muliple instance

    Posted Tue November 27, 2018 03:47 PM
    ​Hi,

    Suddenly our SAP backup which has been configured with Brtool utility on tsm started getting failed I could not see any error on tsm and on client log that doesn't have much information not able to understand how to get it working we have 2 schedules one is full online and another one is incremental since full backup got failed that is why incremental is also failing attaching the error snapshot please help to rescue. 



    ------------------------------
    Prem Singh

    ------------------------------


  • 2.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Wed November 28, 2018 06:08 AM
    Hi All,

    Please help I am unable to find anything to work on.

    Thanks & regards
    Prem Singh Bathyal
    Mobile - 8527337214





  • 3.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Thu November 29, 2018 09:16 AM

    It looks like there are a couple possibilities:

    1. Did a user cancel the backup?
    2. Checkout this link:  http://www.backupcentral.com/forum/5/249593/tsm_for_erp_-_rman_backup_apparently_succeeds__brbackup_give

     

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com





  • 4.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Thu November 29, 2018 10:17 PM
    Hi Brenda,

    Thanks good to see you after a long time in forum you were a life saviour in previous issue, I saw that article earlier.
    But could not reach out to Sap team reason behind it , if from one channel it,s not going through It's trying for another channel. I have 2 suspects in mind one which you indicated someone has cancelled but I don't think that should be the case because I have 10 + database failure on 10 different server, I am presuming no one will cancel on all 10 logically, 

    another point is at the time of booking window so many reclamation is also going on it is also using multiple drives,
    I am suspecting because of that backup is failing because cahnnel is not getting sufficient drive to mount the tape although I have IBM Big lib in place and could not see any detailed report on tsm server for cause of failure.

    Pls share your thoughts as well.


    Thanks & regards
    Prem Singh Bathyal
    Mobile - 8527337214





  • 5.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Fri November 30, 2018 08:54 AM

    Hi Prem,

     

    I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

     

    If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

     

    1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
      1. Query by time period of when it is occurring.
      2. Query by session number to see what detail comes up.
      3. Look for the tape number involved in this backup session.
      4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
    2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
    3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

     

    A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

     

    I am curious what the end result will be for you so let me know what you find.

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com





  • 6.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Fri November 30, 2018 10:30 PM
    Hi Brenda,

    Thanks so much for details knowledge sharing I am on it working on all the suggested way , I will post my findings soon.

    ------------------------------
    Prem Singh

    ------------------------------



  • 7.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Sun December 02, 2018 03:00 PM
    ​Hi Brenda,

    I could see the error log on almost all of the client is as below, after analysing further.

    12/02/2018 18:15:27 ANS1301E Server detected system error
    12/02/2018 18:20:47 ANS1301E Server detected system error
    12/02/2018 18:40:26 ANS1301E Server detected system error
    12/02/2018 19:02:54 ANS1312E Server media mount not possible
    12/02/2018 19:15:43 ANS1301E Server detected system error
    12/02/2018 20:08:26 ANS1312E Server media mount not possible
    12/02/2018 20:33:49 ANS1312E Server media mount not possible

    ------------------------------
    Prem Singh


    ------------------------------



  • 8.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Sun December 02, 2018 03:50 PM
    ​another error I could see on another instance as below and additionally I checked we have enough scratch tapes available and mount is also happening.

    ORA-27028: skgfqcre: sbtbackup returned error
    ORA-19511: Error received from media manager layer, error text:
       BKI7057E: Service open failed because all configured sessions are currently in use.
    channel sbt_2 disabled, job failed on it will be run on another channel

    ------------------------------
    Prem Singh

    ------------------------------



  • 9.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Mon December 03, 2018 09:52 AM

    Prem,

     

    A couple things to check:

     

    Do you know how many mountpoints are requested for each database backup being run?  It is important that you have your maxnummp on each client set to above what they actually use.    If they use 3, make sure the maxnummp is greater than that.

     

    If you are using virtual libraries with Data Domain, etc.  the situation changes dramatically in how you set your maxnummp.  If you are not using that, there is no reason to explain at this point.

     

    Stop you reclamation scripts for temporarily and see if you are experiencing the same issue.

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com





  • 10.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Tue December 25, 2018 11:02 AM
    Edited by Prem Singh Fri June 19, 2020 11:37 AM
    Hi Brenda and All,

    I have worked on it with all the suggested way and thoughts now on one point i have stuck, my most of the backup started working except few here is a example of one wherein database backup is showing completed but arch log backup failing i have tried whatever the best i know and can execute can anyone tell me more where to dig in to resolve it.

    Logs of Arch log backup failure-

    Note- I have changed the original database name with test. 

    BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/cntrltest.dbf
    BR0278E Command output of '/usr/sap/test/SYS/exe/run/backint -u test -f backup -i /oracle/test/saparch/.afacoiba.lst -t file -p /oracle/test/112_64/dbs/inittest.utl -c':
    BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/spfiletest.ora.

    BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/spfiletest.ora
    BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/inittest.sap.

    BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/inittest.sap
    BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/inittest.utl.

    BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/inittest.utl
    BKI1230E: The following file was not processed: /oracle/test/sapreorg/spacetest.log.

    BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/spacetest.log
    BKI1230E: The following file was not processed: /oracle/test/sapreorg/structest.log.

    BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/structest.log
    BKI1230E: The following file was not processed: /oracle/test/saparch/afacoiba.svd.

    BR0233E Backup utility has reported an error while saving file /oracle/test/saparch/afacoiba.svd
    BKI1230E: The following file was not processed: /oracle/test/saparch/archtest.log.

    BR0233E Backup utility has reported an error while saving file /oracle/test/saparch/archtest.log
    BKI1215I: Average transmission rate was 2.091 GB/h (0.594 MB/sec).
    BKI1227I: Average compression factor was 1.000.
    BKI0020I: End of program at: Tue Dec 25 15:24:46 2018 .
    BKI0021I: Elapsed time: 09 min 57 sec .
    BKI0024I: Return code is: 2.

    BR0280I BRARCHIVE time stamp: 2018-12-25 15.24.46
    BR0279E Return code from '/usr/sap/test/SYS/exe/run/backint -u test -f backup -i /oracle/test/saparch/.afacoiba.lst -t file -p /oracle/test/112_64/dbs/inittest.utl -c': 2

    BR0232E 1 of 9 files saved by backup utility
    BR0280I BRARCHIVE time stamp: 2018-12-25 15.24.46
    BR0231E Backup utility call failed


    ------------------------------
    Prem Singh

    ------------------------------



  • 11.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Sat December 29, 2018 01:18 PM
    guys any suggestions, i am struggling . 

    --
    Regards
    Prem Singh Bathyal
    Contact- 8527337214





  • 12.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Sat December 29, 2018 02:35 PM
    After getting more deep dig I can see all the files process backup and deleted while archive backup but at the end below is the file which is creating issue and that is why backup is failing.

    BKI1230E: The following file was not processed: /oracle/PB1/112_64/dbs/spfilePB1.ora



    ------------------------------
    Prem Singh

    ------------------------------



  • 13.  RE: SAP Backup getting failed for multiple Server on muliple instance

    Posted Sat December 29, 2018 03:02 PM
    I am seeing in all the latest archive backup log this error only.

    BKI1208W: The object '/oracle/PB1/112_64/dbs/initPB1.ora' will be retried [2].
    BKI1230E: The following file was not processed: /oracle/PB1/112_64/dbs/initPB1.ora.
    BKI5005E: Tivoli Storage Manager Error.

    looks like it's a permission issue don't know all of a sudden how it is started showing permission issue.

    ------------------------------
    Prem Singh

    ------------------------------