Informix

 View Only
Expand all | Collapse all

Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

  • 1.  Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 02:24 PM

    I am experiencing an issue with an archive Informix database that is stuck in quiescent mode.  When trying to start the service in windows services I get the following error: "The informix IDS – service on  the Local Computer started and stopped. Some services stop automatically if they are not in use by other services or programs". When I run the 'onstat –' command in the Informix terminal I get the following 'IBM Informix Dynamic Server Version 11.50.FC9 – Fast Recovery  <CKPT REQ> -- UP  Blocked CKPT' . It seems that the DB may have been interrupted during a checkpoint and is preventing itself from starting.  I was wondering if the community would happen to know how to approach this issue as I am not very knowledgeable in Informix or what steps need to be done to correct and put the database into standard mode.  I appreciate any help. Thank you



    ------------------------------
    Eric Lovato
    ------------------------------

    #Informix


  • 2.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 02:39 PM
    This could be a number of things, some of them not good.

    But check to make sure that your logical logs are all backed up, as that can cause the engine to block, although I remember this having a different message.

    Run "onstat -l" (that's an "L") and make sure that the flags show U-B and not U--

    Also check the Informix log for more information.  You should be able to see the end of this with onstat -m


    ------------------------------
    Mike Walker
    xDB Systems, Inc
    www.xdbsystems.com
    ------------------------------



  • 3.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 03:21 PM
    Good Afternoon Mike, 
    I was able to run the command on the terminal but i am noticing U___C_L flag listed on the 13th partition and the rest of the partitions being listed as U-- .Thank you for your quick reply


    ------------------------------
    Eric Lovato
    ------------------------------



  • 4.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 03:37 PM
    Okay, this is good.  You need your logical logs to be backed up.  The logical logs are reused but informix cannot use the next logical log because it has not been archived, and if it were overwritten you may not be able to recover if you had to do a restore...so Informix will hang until the log has been backed up.

    I don't know how you normally back up your logs, so you will need to find out why they are not being actively backed up.  Check the informix log for errors.  If you use ontape, check LTAPEDEV in your informix configuration file - perhaps you are writing to a filesystem that has filled, or you are writing to a tape drive that has no tape in it.  Maybe the ontape command needs to be run with -a option, but can only guess at this point.




    ------------------------------
    Mike Walker
    xDB Systems, Inc
    www.xdbsystems.com
    ------------------------------



  • 5.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 03:57 PM
    That is normal. The "L" indicates the logical log that contains the last checkpoint entry and the "C" indicates the logical log that is the one currently being used. The "C"urrent log is normally not backed up. If you run ontape -a without the -d it would prompt you asking whether you want to backup the "C"urrent log. With the -d "N"o is assumed and ontape will not prompt.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 6.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 04:46 PM

    There is a script for backing up the database as follows that uses the ontape -s -L 0 command :

    REM change for your install path
    set INFORMIXDIR=D:\Progra~1\Informix
    REM change if your sever instance name is different that the default
    set INFORMIXSERVER=worx_server
    set ONCONFIG=ONCONFIG.worx_server
    set PATH=D:\Progra~1\Informix\bin;%PATH%
    set CLASSPATH=%INFORMIXDIR%\extend\krakatoa\krakatoa.jar;%INFORMIXDIR%\extend\krakatoa\jdbc.jar;%CLASSPATH%
    set DBTEMP=d:\PROGRA~1\informix\infxtmp
    set CLIENT_LOCALE=EN_US.CP1252
    set DB_LOCALE=EN_US.8859-1
    mode con codepage select=1252

    REM set the path for the logfile
    set LOGFILE=d:\PROGRA~1\informix\level0_backup.log
    cls
    date /T >>%LOGFILE%
    echo "starting level 0 backup" >>%LOGFILE%
    ontape -s -L 0 >>%LOGFILE%
    echo "Completed level 0 backup" >>%LOGFILE%
    date /T >>%LOGFILE%


    I took a look at the level0_backup.log file and the last good backup was done on October 16 however the folder that would contain this backup E:\backups has been cleared presumably by ontape running and not completing. LTAPEDEV variable is set to E:\backups and should be the correct path on a separate windows drive. When I try to run the ontape -s -L 0 command outside of my script I get 'Server is in incompatible state or user authentication failed'. Should i try to execute the ontape command with -a option?

    ------------------------------
    Eric Lovato
    ------------------------------



  • 7.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 05:08 PM
    Eric:

    The ontape -a option only backs up the logical logs. You are saying that your full level 0 archives seem to have been failing. To double check that, first ask the Informix engine:

    onstat -g arc

    Here's output from my server:
    $ onstat -g arc

    IBM Informix Dynamic Server Version 14.10.FC9DE -- On-Line -- Up 2 days 04:40:47 -- 2339972 Kbytes
    2022-11-30 16:57:10


    num   DBSpace            Q Size Q Len  Buffer partnum    size     scanner  

    Dbspaces - Archive Status
    name               number level date             log        log-position
    rootdbs            1      0     11/07/2022.18:12:56 3243       0x2d018    
    llogspace          2      0     11/07/2022.18:12:56 3243       0x2d018    
    plogspace          3      0     11/07/2022.18:12:56 3243       0x2d018    
    datadbs_1          4      0     11/07/2022.18:12:56 3243       0x2d018    
    indexdbs_1         5      0     11/07/2022.18:12:56 3243       0x2d018    
    sbspace            7      0     11/07/2022.18:12:56 3243       0x2d018    
    cdrspace           8      0     11/07/2022.18:12:56 3243       0x2d018    
    indexdbs           9      0     11/07/2022.18:12:56 3243       0x2d018    
    datadbs_2          10     0     11/07/2022.18:12:56 3243       0x2d018    


    So that is saying that the server, and all of its dbspaces. were backed up last on November 7. I don't automatically back up this server as it is used only to test software and Informix. It's only backed up manually about once a month or so. You can see where the backup file(s) are from the last two archives here:

    $ egrep '^TAPEDEV' $INFORMIXDIR/etc/$ONCONFIG
    TAPEDEV /home/art/GoogleDrive/Informix/Archives  
    $ dir /home/art/GoogleDrive/Informix/Archives
    total 6401072
    drwxrwxrwx 2 art      art            4096 Nov  7 18:13 ./
    drwxrwxr-x 4 informix informix       4096 Sep 28 16:27 ../
    -rwxrwxrwx 1 art      art      3277094912 Oct 12 18:32 Elbereth_1_20221012_183241_L0.FC8*
    -rwxrwxrwx 1 art      art      3277586432 Nov  7 18:13 Elbereth_1_L0*

    Also you can look in the message log (parameter MSG) for the last time an archive started and whether it completed successfully or failed and why:

    18:12:56  Level 0 Archive started on rootdbs, llogspace, plogspace, datadbs_1, indexdbs_1, cdrspace, indexdbs, datadbs_2, sbspace
    ...
    18:13:15  Archive on rootdbs, llogspace, plogspace, datadbs_1, indexdbs_1, cdrspace, indexdbs, datadbs_2, sbspace Completed.



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 8.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Thu December 01, 2022 10:32 AM
    Art:
    I ran the onstat -g arc command and received the following:


    I took a look to verfiy the ONCONFIG file and confirmed that the TAPEDEV variable was set to e:\backups



    I then went to verify that there we no files in the backup directory. I took a look at the MSG parameter and reviewed the log file but did not see any information on the backup. Then I remembered we are redirecting the output to a different %LOGFILE% in the script above and have attached the following output

    ------------------------------
    Eric Lovato
    ------------------------------



  • 9.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Thu December 01, 2022 12:03 PM
    Eric:

    And there are no files in E:\backups? That is strange. The last recorded backup according to onstat -g arc was on the 14th but your log files shows completed archives on the 15th and 16th (though the one on the 16th is strange because there are not percent done messages and that's unusual.

    At this point I would open a PMR with IBM and let them sort it out.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 10.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Mon December 19, 2022 11:36 AM
    Thank you, Art and Mike ... I am reaching out to IBM to better understand the PMR process and will be meeting with them today (via webex). I appreciate your time in helping me troubleshoot and I am hoping that it can be resolved

    ------------------------------
    Eric Lovato
    ------------------------------



  • 11.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Thu December 01, 2022 12:18 PM
    Check your informix log to see if it has a message for late on 10/16 or early on 10/17 about a failed or aborted backup.  Maybe it will tell you more information.

    I wonder if you have another script that removes the old backup, or otherwise the failed backup on 10/17 zapped it.  Maybe there is a script that copies the backup somewhere else for safe keeping?

    The bad thing is that it looks like you have no archive available to restore from should you need to...unless it gets copied somewhere else.

    ------------------------------
    Mike Walker
    xDB Systems, Inc
    www.xdbsystems.com
    ------------------------------



  • 12.  RE: Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

    Posted Wed November 30, 2022 05:08 PM
    Yes, try running ontape -a 
    Hopefully this will just back up the logical logs and all will be well.

    However, because the engine is not fully online, I am not sure that it will work.  But try it anyway...

    ------------------------------
    Mike Walker
    xDB Systems, Inc
    www.xdbsystems.com
    ------------------------------