Hi Brenda and All,
I have worked on it with all the suggested way and thoughts now on one point i have stuck, my most of the backup started working except few here is a example of one wherein database backup is showing completed but arch log backup failing i have tried whatever the best i know and can execute can anyone tell me more where to dig in to resolve it.
Logs of Arch log backup failure-
Note- I have changed the original database name with test.
BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/cntrltest.dbf
BR0278E Command output of '/usr/sap/test/SYS/exe/run/backint -u test -f backup -i /oracle/test/saparch/.afacoiba.lst -t file -p /oracle/test/112_64/dbs/inittest.utl -c':
BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/spfiletest.ora.
BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/spfiletest.ora
BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/inittest.sap.
BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/inittest.sap
BKI1230E: The following file was not processed: /oracle/test/112_64/dbs/inittest.utl.
BR0233E Backup utility has reported an error while saving file /oracle/test/112_64/dbs/inittest.utl
BKI1230E: The following file was not processed: /oracle/test/sapreorg/spacetest.log.
BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/spacetest.log
BKI1230E: The following file was not processed: /oracle/test/sapreorg/structest.log.
BR0233E Backup utility has reported an error while saving file /oracle/test/sapreorg/structest.log
BKI1230E: The following file was not processed: /oracle/test/saparch/afacoiba.svd.
BR0233E Backup utility has reported an error while saving file /oracle/test/saparch/afacoiba.svd
BKI1230E: The following file was not processed: /oracle/test/saparch/archtest.log.
BR0233E Backup utility has reported an error while saving file /oracle/test/saparch/archtest.log
BKI1215I: Average transmission rate was 2.091 GB/h (0.594 MB/sec).
BKI1227I: Average compression factor was 1.000.
BKI0020I: End of program at: Tue Dec 25 15:24:46 2018 .
BKI0021I: Elapsed time: 09 min 57 sec .
BKI0024I: Return code is: 2.
BR0280I BRARCHIVE time stamp: 2018-12-25 15.24.46
BR0279E Return code from '/usr/sap/test/SYS/exe/run/backint -u test -f backup -i /oracle/test/saparch/.afacoiba.lst -t file -p /oracle/test/112_64/dbs/inittest.utl -c': 2
BR0232E 1 of 9 files saved by backup utility
BR0280I BRARCHIVE time stamp: 2018-12-25 15.24.46
BR0231E Backup utility call failed
------------------------------
Prem Singh
------------------------------
Original Message:
Sent: 12-03-2018 09:52 AM
From: Brenda Collins
Subject: SAP Backup getting failed for multiple Server on muliple instance
Prem,
A couple things to check:
Do you know how many mountpoints are requested for each database backup being run? It is important that you have your maxnummp on each client set to above what they actually use. If they use 3, make sure the maxnummp is greater than that.
If you are using virtual libraries with Data Domain, etc. the situation changes dramatically in how you set your maxnummp. If you are not using that, there is no reason to explain at this point.
Stop you reclamation scripts for temporarily and see if you are experiencing the same issue.
Brenda Collins,
Sr. Principal Systems Admin | Technical Operations Team
Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112
763.505.6057
Brenda.s.collins@medtronic.com|
medtronic.com | Facebook | LinkedIn | Twitter | YouTube
LET'S TAKE HEALTHCARE
FURTHER, TOGETHER
This message has been marked as Medtronic Controlled
[CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser:
http://emaildisclaimer.medtronic.com
Original Message------
Hi Prem,
I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post. I am still out here though so if you need to reach out to me, you certainly can.
If it is happening on multiple servers at a time, the I would suspect the tape library. It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it. It's hard to pick up on that issue but I have seen it and been through this.
- Query actlog for time when backup is failing. You may have to do this a few different ways to get the right information.
- Query by time period of when it is occurring.
- Query by session number to see what detail comes up.
- Look for the tape number involved in this backup session.
- Then query for that specific tape number over a larger window then what the backup job is running. In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it. (Timing issue) The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first. Tricky to determine but I have seen it.
- Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
- It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing. Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time. You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary. If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives. One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.
A restore could be pre-empting the backup but I am not sure that is the problem. You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.
I am curious what the end result will be for you so let me know what you find.
Brenda Collins,
Sr. Principal Systems Admin | Technical Operations Team
Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112
763.505.6057
Brenda.s.collins@medtronic.com|
medtronic.com | Facebook | LinkedIn | Twitter | YouTube
LET'S TAKE HEALTHCARE
FURTHER, TOGETHER
This message has been marked as Medtronic Controlled
[CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser:
http://emaildisclaimer.medtronic.com