Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Prem,

A couple things to check:

Do you know how many mountpoints are requested for each database backup being run?  It is important that you have your maxnummp on each client set to above what they actually use.    If they use 3, make sure the maxnummp is greater than that.

If you are using virtual libraries with Data Domain, etc.  the situation changes dramatically in how you set your maxnummp.  If you are not using that, there is no reason to explain at this point.

Stop you reclamation scripts for temporarily and see if you are experiencing the same issue.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Prem,

A couple things to check:

Do you know how many mountpoints are requested for each database backup being run?  It is important that you have your maxnummp on each client set to above what they actually use.    If they use 3, make sure the maxnummp is greater than that.

If you are using virtual libraries with Data Domain, etc.  the situation changes dramatically in how you set your maxnummp.  If you are not using that, there is no reason to explain at this point.

Stop you reclamation scripts for temporarily and see if you are experiencing the same issue.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Prem,

A couple things to check:

Do you know how many mountpoints are requested for each database backup being run?  It is important that you have your maxnummp on each client set to above what they actually use.    If they use 3, make sure the maxnummp is greater than that.

If you are using virtual libraries with Data Domain, etc.  the situation changes dramatically in how you set your maxnummp.  If you are not using that, there is no reason to explain at this point.

Stop you reclamation scripts for temporarily and see if you are experiencing the same issue.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled

Hi Prem,

I do see your posts on the community but if there are other people jumping in on discussion, I generally figure you have it handled so I don't take the time to always post.  I am still out here though so if you need to reach out to me, you certainly can.

If it is happening on multiple servers at a time, the I would suspect the tape library.  It might not be so much as you don't have the drives but that the reclamation process has grabbed the tape needed for the backup and is in the process of running reclamation against it.  It's hard to pick up on that issue but I have seen it and been through this.

1. Query actlog for time when backup is failing.  You may have to do this a few different ways to get the right information. 
   1. Query by time period of when it is occurring.
   2. Query by session number to see what detail comes up.
   3. Look for the tape number involved in this backup session.
   4. Then query for that specific tape number over a larger window then what the backup job is running.  In doing this, you are trying to determine if a reclamation process already grabbed that tape after the backup started but before the job requested it.  (Timing issue)  The backup has already determined it needs that specific tape from the point it started but then reclamation sneaks in there and gets the tape mounted first.  Tricky to determine but I have seen it. 
2. Another thing to check is to verify the stgpool the tape exists in and then check to see if reclamation is running during that time period on that same stgpool.
3. It is possible that all your tape drives are in use, if there is some consistency in the timing of when your backups are failing.  Check times of backup failures and maybe set up a simple query to see how many tape drives are in use during that time.  You could create a simple script to show 'q mount' and run every 30 minutes or whatever time period you think is necessary.  If you use that for a couple days, you could probably get a feel for how much utilization is going on with the drives.  One you have determined this, you can turn the script off because you probably wouldn't use it unless you are experiencing an issue.

A restore could be pre-empting the backup but I am not sure that is the problem.  You could check for that in the actlog as well during the time period you are having problems by doing a query for 'restore'.

I am curious what the end result will be for you so let me know what you find.

Brenda Collins,

Sr. Principal Systems Admin | Technical Operations Team

Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

763.505.6057

Brenda.s.collins@medtronic.com|

medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

LET'S TAKE HEALTHCARE
FURTHER, TOGETHER

This message has been marked as Medtronic Controlled