Hi All,
I need your help badly all of sudden one of my tsm server stopped doing anything whenever any backup request is coming to that server we are seeing server detected system error and all the backup's are failing.
we have tsm 6.3.0 on Aix 5. below is the logs which i can see on actlog. Just an addition we have today ran the re balancing as well but nothing helped.
ANR1181E aftxn.c(346): Data storage transaction 0:12254146
was aborted. (SESSION: 6756)
03/13/20 12:07:12 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6801)
03/13/20 12:07:12 ANR1181E aftxn.c(346): Data storage transaction 0:12255212
was aborted. (SESSION: 6801)
03/13/20 12:09:28 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6813)
03/13/20 12:09:28 ANR1181E aftxn.c(346): Data storage transaction 0:12255442
was aborted. (SESSION: 6813)
03/13/20 12:10:27 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6829)
03/13/20 12:10:27 ANR1181E aftxn.c(346): Data storage transaction 0:12255804
was aborted. (SESSION: 6829)
03/13/20 12:10:35 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6543)
03/13/20 12:10:35 ANR1181E aftxn.c(346): Data storage transaction 0:12250436
was aborted. (SESSION: 6543)
03/13/20 12:10:40 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6839)
03/13/20 12:10:40 ANR1181E aftxn.c(346): Data storage transaction 0:12256010
was aborted. (SESSION: 6839)
03/13/20 12:10:51 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6861)
03/13/20 12:10:51 ANR1181E aftxn.c(346): Data storage transaction 0:12256328
was aborted. (SESSION: 6861)
03/13/20 12:11:02 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6866)
03/13/20 12:11:02 ANR1181E aftxn.c(346): Data storage transaction 0:12256400
was aborted. (SESSION: 6866)
03/13/20 12:11:48 ANR2034E SELECT: No match found using this criteria.
(SESSION: 6875)
03/13/20 12:11:59 ANR2034E QUERY DRIVE: No match found using this criteria.
(SESSION: 6878)
03/13/20 12:11:59 ANR2034E SELECT: No match found using this criteria.
Date/Time Message
-------------------- ----------------------------------------------------------
03/13/20 11:30:17 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6478)
03/13/20 11:30:17 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6478)
03/13/20 11:30:23 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6460)
03/13/20 11:30:23 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6460)
03/13/20 11:30:28 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6487)
03/13/20 11:30:28 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6487)
03/13/20 11:30:39 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6498)
03/13/20 11:30:39 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6498)
03/13/20 11:30:50 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6504)
03/13/20 11:30:50 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6504)
03/13/20 11:33:04 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 6470)
03/13/20 11:33:04 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 6470)
03/13/20 11:34:43 ANR0162W Supplemental database diagnostic information:
-1:58030:-1655 ([IBM][CLI Driver][DB2/AIX64] SQL1655C
The operation could not be completed due to an error
accessing data on disk. SQLSTATE=58030). (SESSION: 5656)
03/13/20 11:34:43 ANR0102E afcreate.c(2701): Error 9999 inserting row in
table "AF.Segments". (SESSION: 5656)
03/13/20 11:34:43 ANE4994S (Session: 5656, Node: FRMRS162_ORA) TDP Oracle
HP ANU0599 TDP for Oracle: (20130): => (FRMRS162_ORA)
ANS1301E (RC1) Server detected system error(SESSION:
------------------------------
Thanks
Prem
------------------------------