Data Protection Software

Expand all | Collapse all

TSM Server Detected System Error

  • 1.  TSM Server Detected System Error

    Posted Fri March 13, 2020 07:30 AM
    Hi All,

    I need your help badly all of sudden one of my tsm server stopped doing anything whenever any backup request is coming to that server we are seeing server detected system error and all the backup's are failing.
    we have tsm 6.3.0 on Aix 5. below is the logs which i can see on actlog. Just an addition we have today ran the re balancing as well but nothing helped.

    ANR1181E aftxn.c(346): Data storage transaction 0:12254146
    was aborted. (SESSION: 6756)
    03/13/20 12:07:12 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6801)
    03/13/20 12:07:12 ANR1181E aftxn.c(346): Data storage transaction 0:12255212
    was aborted. (SESSION: 6801)
    03/13/20 12:09:28 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6813)
    03/13/20 12:09:28 ANR1181E aftxn.c(346): Data storage transaction 0:12255442
    was aborted. (SESSION: 6813)
    03/13/20 12:10:27 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6829)
    03/13/20 12:10:27 ANR1181E aftxn.c(346): Data storage transaction 0:12255804
    was aborted. (SESSION: 6829)
    03/13/20 12:10:35 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6543)
    03/13/20 12:10:35 ANR1181E aftxn.c(346): Data storage transaction 0:12250436
    was aborted. (SESSION: 6543)
    03/13/20 12:10:40 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6839)
    03/13/20 12:10:40 ANR1181E aftxn.c(346): Data storage transaction 0:12256010
    was aborted. (SESSION: 6839)
    03/13/20 12:10:51 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6861)
    03/13/20 12:10:51 ANR1181E aftxn.c(346): Data storage transaction 0:12256328
    was aborted. (SESSION: 6861)
    03/13/20 12:11:02 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6866)
    03/13/20 12:11:02 ANR1181E aftxn.c(346): Data storage transaction 0:12256400
    was aborted. (SESSION: 6866)
    03/13/20 12:11:48 ANR2034E SELECT: No match found using this criteria.
    (SESSION: 6875)
    03/13/20 12:11:59 ANR2034E QUERY DRIVE: No match found using this criteria.
    (SESSION: 6878)
    03/13/20 12:11:59 ANR2034E SELECT: No match found using this criteria.

    Date/Time Message
    -------------------- ----------------------------------------------------------
    03/13/20 11:30:17 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6478)
    03/13/20 11:30:17 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6478)
    03/13/20 11:30:23 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6460)
    03/13/20 11:30:23 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6460)
    03/13/20 11:30:28 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6487)
    03/13/20 11:30:28 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6487)
    03/13/20 11:30:39 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6498)
    03/13/20 11:30:39 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6498)
    03/13/20 11:30:50 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6504)
    03/13/20 11:30:50 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6504)
    03/13/20 11:33:04 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 6470)
    03/13/20 11:33:04 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 6470)
    03/13/20 11:34:43 ANR0162W Supplemental database diagnostic information:
    -1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
    The operation could not be completed due to an error
    accessing data on disk. SQLSTATE=58030). (SESSION: 5656)
    03/13/20 11:34:43 ANR0102E afcreate.c(2701): Error 9999 inserting row in
    table "AF.Segments". (SESSION: 5656)
    03/13/20 11:34:43 ANE4994S (Session: 5656, Node: FRMRS162_ORA) TDP Oracle
    HP ANU0599 TDP for Oracle: (20130): => (FRMRS162_ORA)
    ANS1301E (RC1) Server detected system error(SESSION:



    ------------------------------
    Thanks
    Prem

    ------------------------------


  • 2.  RE: TSM Server Detected System Error

    Posted Fri March 13, 2020 09:40 AM
    Guys,

    Do let me know if anyone has any suggestion.

    Thanks
    Prem





  • 3.  RE: TSM Server Detected System Error

    Posted Mon March 16, 2020 03:58 AM
    Anyone has any clue .


    Thanks
    Prem





  • 4.  RE: TSM Server Detected System Error

    Posted Mon March 16, 2020 05:47 AM
    Hi Prem,

    Are all of your volume groups varied on, and all disks marked available?

    It may also be useful to check the db2diag.log for and database related errors.

    Regards
    Darren

    ------------------------------
    Darren Sanders
    Technical Architect - Data Protection & Storage
    Celerity Ltd

    http://www.celerity-uk.com
    www.linkedin.com/in/darren-sanders-a8770a4
    #ibmchampion storage
    ------------------------------



  • 5.  RE: TSM Server Detected System Error

    Posted Mon March 16, 2020 07:37 AM
    Hi Darren,

    Thanks for the response all the disk volumes are online however whenever I check on db2diag it doesn't give me specific clue but on instance directory 100's TSM bad page folder's and files are creating.

    Thanks
    Prem





  • 6.  RE: TSM Server Detected System Error

    Posted Wed March 18, 2020 05:39 AM
    Edited by Jochen Meixner Fri June 19, 2020 11:36 AM
    Hi Prem,
    this looks like a corruption of indices in DB2.
    Without the db2diag is it hard to tell which problem it is.
    DB2 problem IT05944 is about a smilar problem. I could not reach the linked IBM site, the following site shows the problem desciption:
    https://www.cursor-distribution.de/en/component/ibmbuglist/?view=bugdetail&get_bug=IT05944
    I hope this helps to clarify if this is the problem. 

    Best regards,
    Jochen

    ------------------------------
    Jochen Meixner
    Principal Consultant Storage & Backup Solutions
    CONET Services GmbH
    Duesseldorf, Germany
    ------------------------------