Data Protection Software

Expand all | Collapse all

TSM Server intance is not coming up

  • 1.  TSM Server intance is not coming up

    Posted Mon December 03, 2018 05:02 AM
    Hi All,
    All of sudden today I notice one of my tsm instance is went down,it's s not coming up. When we try to restart the tsm server from startserver script it shows it is started which we can see on services as well via ps -ef |grep dsm but within a minute it is going off I could see only this log.
    Asynch I/o failed I could see so many db2 dump error all of sudden that piled up the file system space of home directory later on we have clear but still on the same stage.

    Aix version- 6
    TSM- 7.1.7

    ------------------------------
    Prem Singh


    ------------------------------


  • 2.  RE: TSM Server intance is not coming up

    Posted Mon December 03, 2018 02:31 PM
    ​Hi All,

    This is the error finally we got .

    db2start
    12/03/2018 21:01:29     0   0   SQL1063N  DB2START processing was successful.
    SQL1063N  DB2START processing was successful.
    db2 get db cfg
    ksh: connect:  not found.
    $ db2 get db cfg
    SQL1024N  A database connection does not exist.  SQLSTATE=08003
    $ db2 connect to tsmdb1
    SQL0293N  Error accessing a table space container.  SQLSTATE=57048.
    It is showing closed on 2 mount points not able to understand how to set in open/syncd state.



    ------------------------------
    Prem Singh

    ------------------------------



  • 3.  RE: TSM Server intance is not coming up

    Posted Tue December 04, 2018 08:17 AM
    If it's AIX, run fsck -y /mountpoint and notice messages.
    Run this twice, second time should be clean.
    After that mount /mountpoint and verify they are mounted.
    If that doesn't work you'll have to dig some more into the errors reported.
    Check your errpt -a | more to see what messages were reported on boot.
    I suspect if your server went down hard these filesystems are just dirty.
    Your not going to get DB2 online until all dbspaces are available.

    ------------------------------
    Jason Buhr
    Systems Engineer Architect
    SpartanNash
    Edina MN
    952-897-8104
    ------------------------------



  • 4.  RE: TSM Server intance is not coming up

    Posted Tue December 04, 2018 12:33 PM
    Thanks Jason,

    I tried that but unfortunately it went down because of capacity crunch on storage it was thin provisioned and already over commit that is why it is dismounted now Storage team has given this solution you please share your thoughts as well.

    VG_Name

    HDISK

    Size in MB

    Serial_Number

    Status

    dbrepstgpoolvg

    hdisk44

    0

    6005076300810B75980000000000002A

    Missing

    dbrepstgpoolvg

    hdisk53

    0

    6005076300810B759800000000000033

    Missing

    dbstgpool01vg

    hdisk38

    2097152

    6005076300810B759800000000000024

    VarriedOFF

    dbstgpool01vg

    hdisk39

    2097152

    6005076300810B759800000000000025

    VarriedOFF

    dbstgpool01vg

    hdisk46

    0

    6005076300810B75980000000000002C

    VarriedOFF

    dbstgpool01vg

    hdisk55

    0

    6005076300810B759800000000000036

    VarriedOFF

    dbstgpool01vg

    hdisk56

    0

    6005076300810B759800000000000037

    VarriedOFF

    fsrepstgpoolvg

    hdisk54

    0

    6005076300810B759800000000000034

    Missing

    tsmdbvg

    hdisk47

    0

    6005076300810B75980000000000002D

    Missing

     

    These VG's are mounted on the AIX filesystems and used for storing Backed up data and TSM Servers

    DB2 Database (tsmdbvg) . As these Mount points are missing due to the Space issue on V5000 Storage Array in R&D environment due to the Thin Provisioned LUN's are already over occupied.

     

    Resolution Plan:-

    As we don't have any space left out in the IBM-v5000 storage and right now we can't procure the new storage for the array and can't add a new enclosure in the rack, so storage team have suggested following plan and workaround.

     

    1)      Storage Team will create a new LUN with 6x2TB on another IBM-v7000 and zone it to the server.

    2)      Unix Team will create a new Volume group on the Server and migrate the data of following old VG's to new VG's and sync the data.

    fsstgpoolvg

    hdisk25

    2097152

    6005076300810B759800000000000017

    OK

    fsstgpoolvg

    hdisk26

    2097152

    6005076300810B759800000000000018

    OK

    fsstgpoolvg

    hdisk27

    2097152

    6005076300810B759800000000000019

    OK

    fsstgpoolvg

    hdisk28

    2097152

    6005076300810B75980000000000001A

    OK

    fsstgpoolvg

    hdisk29

    2097152

    6005076300810B75980000000000001B

    OK

    fsstgpoolvg

    hdisk30

    2097152

    6005076300810B75980000000000001C

    OK

     

     

     

    3)      After confirmation of data migration from UNIX team, Storage Team will delete the old LUN's and from the storage Array v5000.

    hdisk25

    hdisk26

    hdisk27

    hdisk28

    hdisk29

    hdisk30

     

     

     

     

     

     

     

     

     

     

     

    4)      As the Older LUN's are Thin provisioned, following missing LUN's can be online now and should be able to mount.

    VG_Name

    HDISK

    Size in MB

    Serial_Number

    Status

    dbrepstgpoolvg

    hdisk44

    0

    6005076300810B75980000000000002A

    Missing

    dbrepstgpoolvg

    hdisk53

    0

    6005076300810B759800000000000033

    Missing

    fsrepstgpoolvg

    hdisk54

    0

    6005076300810B759800000000000034

    Missing

    tsmdbvg

    hdisk47

    0

    6005076300810B75980000000000002D

    Missing

     

    5)      After the Mounts points are Online, Backup Team will make the DB2 and TSM Services UP. The TSM database should be UP and running.

    6)      After which the backup storage pools will be configured and routed to newly assigned LUN's from IBM-v7000.


    --
    Regards
    Prem Singh Bathyal
    Contact- 8527337214