IBM Storage Defender

IBM Storage Defender

Early threat detection and secure data recovery

 View Only
Back to discussions
Expand all | Collapse all

Database backup stuck on 80%

  • 1.  Database backup stuck on 80%

    Posted 04/02/19 11:56 PM
    ​Hi All,

    I am facing issue on TSM Server which is hosted on Aix 6 and tsm version is 5.5.1 it is rdbms database not able to find a clue where it's going wrong all of sudden from last 6-7 days my database backup is not working it stopped after completing 80% it completed hung the tsm server and as well as AIX host after that I am just restarting the host and all is normal but nothing is sorted out any new idea and suggestion is highly appreciated. below is the mount point status.

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
    /dev/hd4           0.25      0.07   71%     9169    32% /
    /dev/hd2           2.75      0.51   82%    38298    23% /usr
    /dev/hd9var        1.62      0.05   97%    12026    47% /var
    /dev/hd3           2.12      0.69   68%    10009     6% /tmp
    /dev/fslv04        0.06      0.00  100%       64   100% /AIXstats
    /dev/fslv08        0.62      0.03   96%    14979    71% /TSMSTATS
    /dev/hd1           0.03      0.01   56%     1746    32% /home
    /dev/hd10opt       2.66      0.56   79%    17674    12% /opt
    /proc                 -         -    -         -     -  /proc
    /dev/tnglv         0.25      0.05   80%      822     7% /tng
    /dev/fslv05       14.75     12.79   14%        6     1% /tsm4/db
    /dev/fslv06        5.50      1.50   73%        5     1% /tsm4/log1
    /dev/fslv07        5.00      1.00   81%        5     1% /tsm4/log2
    /dev/fslv02      374.31     17.81   96%       41     1% /tsm40/db
    /dev/fslv01       13.97      0.99   93%        8     1% /tsm40/log1
    /dev/fslv03       13.97      0.99   93%        8     1% /tsm40/log2
    /dev/dsmserv_lv      2.06      0.05   98%      465     4% /usr/tivoli
    /dev/fslv00        1.03      0.62   40%      484     1% /usr/tivoli_rootvg
    /dev/hd11admin      0.12      0.12    1%        4     1% /admin

    ------------------------------
    Prem Singh

    ------------------------------


  • 2.  RE: Database backup stuck on 80%

    Posted 04/03/19 12:21 AM

    show logpinned
    Dirty page Lsn=57179491.239.2399, Last DB backup Lsn=57179492.31.3048, Transaction table Lsn=0.0.0, Running DB backup Lsn=0.0.0, Log truncation Lsn=57179491.89.4053

    Lsn=57179491.89.4053, Owner=DB, Length=238
    Type=Update, Flags=C2, Action=ExtInsert, Page=4690610, Tsn=0:875981766, PrevLsn=0.0.0, UndoNextLsn=0.0.0, UpdtLsn=57179491.54.127 ===> ObjName=AS.Segments, Index=14,
    RootAddr=15, PartKeyLen=4, NonPartKeyLen=4, DataLen=130
    The recovery log is not pinned.  Space in the recovery log will be freed at the next checkpoint

    --

    Regards
    Prem



    ------Original Message------

    ​Hi All,

    I am facing issue on TSM Server which is hosted on Aix 6 and tsm version is 5.5.1 it is rdbms database not able to find a clue where it's going wrong all of sudden from last 6-7 days my database backup is not working it stopped after completing 80% it completed hung the tsm server and as well as AIX host after that I am just restarting the host and all is normal but nothing is sorted out any new idea and suggestion is highly appreciated. below is the mount point status.

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
    /dev/hd4           0.25      0.07   71%     9169    32% /
    /dev/hd2           2.75      0.51   82%    38298    23% /usr
    /dev/hd9var        1.62      0.05   97%    12026    47% /var
    /dev/hd3           2.12      0.69   68%    10009     6% /tmp
    /dev/fslv04        0.06      0.00  100%       64   100% /AIXstats
    /dev/fslv08        0.62      0.03   96%    14979    71% /TSMSTATS
    /dev/hd1           0.03      0.01   56%     1746    32% /home
    /dev/hd10opt       2.66      0.56   79%    17674    12% /opt
    /proc                 -         -    -         -     -  /proc
    /dev/tnglv         0.25      0.05   80%      822     7% /tng
    /dev/fslv05       14.75     12.79   14%        6     1% /tsm4/db
    /dev/fslv06        5.50      1.50   73%        5     1% /tsm4/log1
    /dev/fslv07        5.00      1.00   81%        5     1% /tsm4/log2
    /dev/fslv02      374.31     17.81   96%       41     1% /tsm40/db
    /dev/fslv01       13.97      0.99   93%        8     1% /tsm40/log1
    /dev/fslv03       13.97      0.99   93%        8     1% /tsm40/log2
    /dev/dsmserv_lv      2.06      0.05   98%      465     4% /usr/tivoli
    /dev/fslv00        1.03      0.62   40%      484     1% /usr/tivoli_rootvg
    /dev/hd11admin      0.12      0.12    1%        4     1% /admin

    ------------------------------
    Prem Singh

    ------------------------------


  • 3.  RE: Database backup stuck on 80%

    Posted 04/03/19 09:12 AM
    Any suggestion any advise please.

    ------------------------------
    Prem Singh

    ------------------------------

    Original Message


  • 4.  RE: Database backup stuck on 80%

    Posted 04/03/19 09:48 AM
    Hi Prem,
    Can you check the size of your database and log space available?  Is it possible for you to increase the amount of space available to both? If you can increase the space, then try the db backup again and see if it runs all the way through.   Let me know if that works.​

    ------------------------------
    Brenda Collins
    Sr. Principal Storage Admin
    Medtronic
    Moundsview MN
    ------------------------------



  • 5.  RE: Database backup stuck on 80%

    Posted 04/03/19 10:49 AM
    ​Prem, 
    Please send me a ' q db f=d' and a 'q log f=d'.  Do you still have the version 5 TSM manuals?  I don't have mine anymore but there is a limit to the database and log sizes that you are able to manage on the system as well. 

    From what I see in looking at the details you sent me, you are at risk of losing your system completely.  Most filesystems will start causing problems when they are in the 90-100% range.  If this system cannot have additional disk added from a SAN environment to supplement the physical disk on the system, it will crash at some point.

    The other issue which may or may not be a problem is the size of your TSM database.  There is a limit to how large of a database that version 5 will support.  I don't recall what that is but if you have the old manuals from version 5, you should be able to find that information.  

    The data you sent me is below, I just wanted to post it so others could see and possibly add their comments and/or solutions:

    yes correct it is rdbms on tsm 5.5 here is the output from host, where i can see i have space available, it is not completely occupied, frankly speaking now i can't increase the space, we don't have space available on storage end.

     

    frmrsstsm40:/ # df -g /tsm40/db

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv02      374.31     17.81   96%       41     1% /tsm40/db

    frmrsstsm40:/ # df -g /tsm40/log1

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv01       13.97      0.99   93%        8     1% /tsm40/log1

    frmrsstsm40:/ # df -g /tsm40/log2

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv03       13.97      0.99   93%        8     1% /tsm40/log2

     



    ------------------------------
    Brenda Collins
    Sr. Principal Storage Admin
    Medtronic
    Moundsview MN
    ------------------------------

    Original Message


  • 6.  RE: Database backup stuck on 80%

    Posted 04/03/19 11:00 AM


    AdminPrem <premsingh17@gmail.com>

    Attachments8:21 PM (5 minutes ago)
    to Brenda
    q db and log f=d as of now can't be provided because server is not responding as of now VG is locked I need to take approval for rebooting the host.


    Yes you are right I am not only on risk for me this server will not function anymore I have to restore the database on last database backup date which is 15 days prior to that, because Database size limitation is 530 GB and log limitations is 13 GB as of now i can see log size is already 13 GB so it can'be extended, we can add one more volume into the log and extend that if we will get the space then only, I am attaching the manual of DB and log limitation.

    Suggest me one thing /tsm40/log2, /tsm40/log1 I have few files which can be moved which is very older I am referring to below one. i will take db backup and can move once again although looks impractical for me .
    frmrsstsm40:/tsm40/log1 # ls -lrt
    total 27222016
    drwxr-xr-x    2 root     system          256 May 31 2007  lost+found
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:55 log03.dsm
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:55 log02.dsm
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:56 log01.dsm
    -rw-r--r--    1 root     system   1049624576 Apr 03 13:16 log04.dsm
    frmrsstsm40:/tsm40/log1 # cd /tsm40/log2
    frmrsstsm40:/tsm40/log2 # ls -lrt
    total 27222016
    drwxr-xr-x    2 root     system          256 May 31 2007  lost+found
    -rw-r--r--    1 root     system   1049624576 Dec 27 2017  log04.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:16 log03.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:16 log02.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:33 log01.dsm
    frmrsstsm40:/tsm40/log2 #


    --
    Regards
    Prem


    ------Original Message------

    ​Prem, 
    Please send me a ' q db f=d' and a 'q log f=d'.  Do you still have the version 5 TSM manuals?  I don't have mine anymore but there is a limit to the database and log sizes that you are able to manage on the system as well. 

    From what I see in looking at the details you sent me, you are at risk of losing your system completely.  Most filesystems will start causing problems when they are in the 90-100% range.  If this system cannot have additional disk added from a SAN environment to supplement the physical disk on the system, it will crash at some point.

    The other issue which may or may not be a problem is the size of your TSM database.  There is a limit to how large of a database that version 5 will support.  I don't recall what that is but if you have the old manuals from version 5, you should be able to find that information.  

    The data you sent me is below, I just wanted to post it so others could see and possibly add their comments and/or solutions:

    yes correct it is rdbms on tsm 5.5 here is the output from host, where i can see i have space available, it is not completely occupied, frankly speaking now i can't increase the space, we don't have space available on storage end.

     

    frmrsstsm40:/ # df -g /tsm40/db

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv02      374.31     17.81   96%       41     1% /tsm40/db

    frmrsstsm40:/ # df -g /tsm40/log1

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv01       13.97      0.99   93%        8     1% /tsm40/log1

    frmrsstsm40:/ # df -g /tsm40/log2

    Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

    /dev/fslv03       13.97      0.99   93%        8     1% /tsm40/log2

     



    ------------------------------
    Brenda Collins
    Sr. Principal Storage Admin
    Medtronic
    Moundsview MN
    ------------------------------

    Attachment(s)

    pdf
    db_recovery_log.pdf   1.34 MB 1 version


  • 7.  RE: Database backup stuck on 80%

    Posted 04/03/19 11:14 AM
    ​Can anyone else jump in here and offer some advice to help Prem out of this situation please?

    ------------------------------
    Brenda Collins
    Sr. Principal Storage Admin
    Medtronic
    Moundsview MN
    ------------------------------

    Original Message


  • 8.  RE: Database backup stuck on 80%

    Posted 04/03/19 11:30 AM

    It seems like you should be able to remove one of the older ones but I am not sure what deleting that will do when bringing the system back up.  Can you run the reduce log command or is the system totally hung up?  I feel like it is more of an aix issue at this point than possibly a tsm issue.  Are you running the database in normal mode vs. rollforward mode?  If I remember right, we used to have issues with the rollforward mode due to it needing more space in the way it functions.

     

    I am unsure if you delete one of those log volumes if it would recognize the increased space or not vs. running the reduce log command. It does look like those logs you mention are the oldest.  I am worried if I give you the wrong information, it might crash your server.

     

    Use the reduce log command to decrease the amount of space that can be used by the recovery log. The log must be in normal mode.

    When reducing the capacity of the database or recovery log, you must reduce by increments of 4MB. If you do not specify the reduction in 4MB increments, Tivoli Storage Manager rounds the number to the next 4MB partition.

    This message has been marked as Medtronic Controlled

     



    ------Original Message------



    AdminPrem <premsingh17@gmail.com>

    Attachments8:21 PM (5 minutes ago)
    to Brenda
    q db and log f=d as of now can't be provided because server is not responding as of now VG is locked I need to take approval for rebooting the host.


    Yes you are right I am not only on risk for me this server will not function anymore I have to restore the database on last database backup date which is 15 days prior to that, because Database size limitation is 530 GB and log limitations is 13 GB as of now i can see log size is already 13 GB so it can'be extended, we can add one more volume into the log and extend that if we will get the space then only, I am attaching the manual of DB and log limitation.

    Suggest me one thing /tsm40/log2, /tsm40/log1 I have few files which can be moved which is very older I am referring to below one. i will take db backup and can move once again although looks impractical for me .
    frmrsstsm40:/tsm40/log1 # ls -lrt
    total 27222016
    drwxr-xr-x    2 root     system          256 May 31 2007  lost+found
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:55 log03.dsm
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:55 log02.dsm
    -rw-r--r--    1 root     system   4296015872 Oct 12 21:56 log01.dsm
    -rw-r--r--    1 root     system   1049624576 Apr 03 13:16 log04.dsm
    frmrsstsm40:/tsm40/log1 # cd /tsm40/log2
    frmrsstsm40:/tsm40/log2 # ls -lrt
    total 27222016
    drwxr-xr-x    2 root     system          256 May 31 2007  lost+found
    -rw-r--r--    1 root     system   1049624576 Dec 27 2017  log04.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:16 log03.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:16 log02.dsm
    -rw-r--r--    1 root     system   4296015872 Apr 03 13:33 log01.dsm
    frmrsstsm40:/tsm40/log2 #


    --
    Regards
    Prem




  • 9.  RE: Database backup stuck on 80%

    Posted 04/03/19 01:36 PM
    Hi Brenda,

    Thank you for continuous help, I am collecting more logs just to make sure everything will go smoothly I will give it a try tomorrow I have response awaited from AIX expert as well, I will let you know and update all the results tomorrow.

    Thanks 
    Prem


    ------Original Message------

    It seems like you should be able to remove one of the older ones but I am not sure what deleting that will do when bringing the system back up.  Can you run the reduce log command or is the system totally hung up?  I feel like it is more of an aix issue at this point than possibly a tsm issue.  Are you running the database in normal mode vs. rollforward mode?  If I remember right, we used to have issues with the rollforward mode due to it needing more space in the way it functions.

     

    I am unsure if you delete one of those log volumes if it would recognize the increased space or not vs. running the reduce log command. It does look like those logs you mention are the oldest.  I am worried if I give you the wrong information, it might crash your server.

     

    Use the reduce log command to decrease the amount of space that can be used by the recovery log. The log must be in normal mode.

    When reducing the capacity of the database or recovery log, you must reduce by increments of 4MB. If you do not specify the reduction in 4MB increments, Tivoli Storage Manager rounds the number to the next 4MB partition.

    This message has been marked as Medtronic Controlled

     





  • 10.  RE: Database backup stuck on 80%

    Posted 04/03/19 03:05 PM

    Thanks for the update!  I hope all goes well.

     



    ------Original Message------

    Hi Brenda,

    Thank you for continuous help, I am collecting more logs just to make sure everything will go smoothly I will give it a try tomorrow I have response awaited from AIX expert as well, I will let you know and update all the results tomorrow.

    Thanks 
    Prem




  • 11.  RE: Database backup stuck on 80%

    Posted 04/04/19 01:36 PM
    Hi Brenda and All,

    I manage to switch on the server but not attempted database backup and unfortunately we don't have space available on storage side.

    But I have collected the output of q db and q log as below if anyone find any clue let me know.

    tsm: TSM40>q db f=d

     

    Available  Assigned    Maximum    Maximum     Page      Total       Used    Pct   Max.  Physical  Buffer        Total   Cache   Cache  Backup in     Type of  Incrementals  Changed  Percentage       Last  Estimate of  Last Estim-

        Space  Capacity  Extension  Reduction     Size     Usable      Pages   Util    Pct   Volumes    Pool       Buffer     Hit    Wait  Progress?   Backup In    Since Last    Since     Changed   Complete  Recoverable       ate of

         (MB)      (MB)       (MB)       (MB)  (bytes)      Pages                     Util             Pages     Requests    Pct.    Pct.               Progress          Full     Last                 Backup   Space (MB)  Recoverable

                                                                                                                                                                                 Backup              Date/Time                Space (MB)

                                                                                                                                                                                   (MB)

    ---------  --------  ---------  ---------  -------  ---------  ---------  -----  -----  --------  ------  -----------  ------  ------  ---------  ----------  ------------  -------  ----------  ---------  -----------  -----------

      364,960   364,960          0     20,396    4,096  93,429,76  28,287,67   30.3   30.3        37  287,14   35,185,740   95.83    0.00         No                         0  5,203.6        4.71  03/21/19

                                                                0          0                               4                                                                          9                18:18:-

                                                                                                                                                                                                            51

     

    tsm: TSM40>q log f=d

     

    Available     Assigned       Maximum       Maximum        Page         Total          Used       Pct      Max.     Physical        Log        Log        Log      Cumulative     Consumption

        Space     Capacity     Extension     Reduction        Size        Usable         Pages      Util       Pct      Volumes       Pool       Pool       Pool     Consumption           Reset

         (MB)         (MB)          (MB)          (MB)     (bytes)         Pages                              Util                   Pages       Pct.       Pct.            (MB)       Date/Time

                                                                                                                                                 Util       Wait

    ---------     --------     ---------     ---------     -------     ---------     ---------     -----     -----     --------     ------     ------     ------     -----------     -----------

       13,288       13,288             0        13,284       4,096     3,401,216           337       0.0       5.2            7        512       1.13       0.00     6,851,965.8      12/28/17

                                                                                                                                                                               0        00:42:48

     


    Thanks 
    Prem 


    ------Original Message------

    Thanks for the update!  I hope all goes well.

     





  • 12.  RE: Database backup stuck on 80%

    Posted 04/04/19 03:25 PM

    Is your database set in normal mode or rollforward?

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com


    ------Original Message------

    Hi Brenda and All,

    I manage to switch on the server but not attempted database backup and unfortunately we don't have space available on storage side.

    But I have collected the output of q db and q log as below if anyone find any clue let me know.

    tsm: TSM40>q db f=d

     

    Available  Assigned    Maximum    Maximum     Page      Total       Used    Pct   Max.  Physical  Buffer        Total   Cache   Cache  Backup in     Type of  Incrementals  Changed  Percentage       Last  Estimate of  Last Estim-

        Space  Capacity  Extension  Reduction     Size     Usable      Pages   Util    Pct   Volumes    Pool       Buffer     Hit    Wait  Progress?   Backup In    Since Last    Since     Changed   Complete  Recoverable       ate of

         (MB)      (MB)       (MB)       (MB)  (bytes)      Pages                     Util             Pages     Requests    Pct.    Pct.               Progress          Full     Last                 Backup   Space (MB)  Recoverable

                                                                                                                                                                                 Backup              Date/Time                Space (MB)

                                                                                                                                                                                   (MB)

    ---------  --------  ---------  ---------  -------  ---------  ---------  -----  -----  --------  ------  -----------  ------  ------  ---------  ----------  ------------  -------  ----------  ---------  -----------  -----------

      364,960   364,960          0     20,396    4,096  93,429,76  28,287,67   30.3   30.3        37  287,14   35,185,740   95.83    0.00         No                         0  5,203.6        4.71  03/21/19

                                                                0          0                               4                                                                          9                18:18:-

                                                                                                                                                                                                            51

     

    tsm: TSM40>q log f=d

     

    Available     Assigned       Maximum       Maximum        Page         Total          Used       Pct      Max.     Physical        Log        Log        Log      Cumulative     Consumption

        Space     Capacity     Extension     Reduction        Size        Usable         Pages      Util       Pct      Volumes       Pool       Pool       Pool     Consumption           Reset

         (MB)         (MB)          (MB)          (MB)     (bytes)         Pages                              Util                   Pages       Pct.       Pct.            (MB)       Date/Time

                                                                                                                                                 Util       Wait

    ---------     --------     ---------     ---------     -------     ---------     ---------     -----     -----     --------     ------     ------     ------     -----------     -----------

       13,288       13,288             0        13,284       4,096     3,401,216           337       0.0       5.2            7        512       1.13       0.00     6,851,965.8      12/28/17

                                                                                                                                                                               0        00:42:48

     


    Thanks 
    Prem 




  • 13.  RE: Database backup stuck on 80%

    Posted 04/04/19 10:01 PM
    It is on Normal Mode.

    Thanks
    Prem 


    ------Original Message------

    Is your database set in normal mode or rollforward?

     

    Brenda Collins,

    Sr. Principal Systems Admin | Technical Operations Team

    Medtronic, Inc. | MVC63 | 8200 Coral Sea Street NE | Mounds View, MN 55112

    763.505.6057

    Brenda.s.collins@medtronic.com|

    medtronic.com  |  Facebook  |  LinkedIn  |  Twitter  |  YouTube

    LET'S TAKE HEALTHCARE
    FURTHER, TOGETHER

    Spectrum+Protect+V8_1

    This message has been marked as Medtronic Controlled

     

    [CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is proprietary to Medtronic and is intended for use only by the individual or entity to which it is addressed, and may contain information that is private, privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please delete this mail from your records. To view this notice in other languages you can either select the following link or manually copy and paste the link into the address bar of a web browser: http://emaildisclaimer.medtronic.com




  • 14.  RE: Database backup stuck on 80%

    Posted 04/06/19 07:41 AM
    Guys ,

    Any suggestions, please advise your help is highly appreciated , i am stuck in the middle.

    Thanks 
    Prem 


    ------Original Message------

    It is on Normal Mode.

    Thanks
    Prem 




  • 15.  RE: Database backup stuck on 80%

    Posted 04/10/19 03:42 AM
    Hi Brenda and Everyone,

    I am able to resolve that now instance is up and running without any issue and database backup also completed.

    Thanks you everyone for the help and contribution.

    Thanks 
    Prem 


    ------Original Message------

    Guys ,

    Any suggestions, please advise your help is highly appreciated , i am stuck in the middle.

    Thanks 
    Prem 




Related Content