Informix

 View Only
Expand all | Collapse all

Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

  • 1.  Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Thu February 08, 2024 10:49 AM

    IDS 12.10.FC13 to 14.10.FC10

    NAME="SLES"  VERSION="15-SP5"

     

    Our ERP finally released IDS14 for us to upgrade to.  I ran the upgrade on our test/dev vm and the upgrade went well.  However the logical log backups are just acting weird.  We use the alarmprogram (BACKUP_CMD="ontape -a -d") to "/backups" directory.  Sometimes it seems to do the backup and other times it seems to just be waiting.  Granted not a lot of activity is happened currently on the test/dev so don't expect lots of logs. 

     

    I'm getting emails from IDS with subject of Logs at 81.5% full or whatever the % is and body of

     

    Hello,

      The logical logs on the onconf.cars OnLine Engine are

      at 81.5% full.  Please take appropriate steps.

     

    Thank You

     

    I'm assuming IDS is generating these as not sure what else would.

     

    We have a mixture of buffered and unbuffered databases.

    adamski cars: echo "select name[1,20], is_logging, is_buff_log from sysdatabases" | dbaccess sysmaster

     

    Database selected.

     

     

     

    name                  is_logging is_buff_log

     

    sysmaster                      1           0

    sysutils                       1           0

    sysuser                        1           0

    sysadmin                       1           0

    uni93                          1           0

    cars_audit                     1           0

    cars                           1           0

    uni97                          1           1

    train                          1           1

    uni103                         1           1

     

    10 row(s) retrieved.

     

     

    Snip-it from onconf

     

    ALARMPROGRAM            $INFORMIXDIR/etc/alarmprogram.sh

    ALRM_ALL_EVENTS         0

    STORAGE_FULL_ALARM      600,3

    SYSALARMPROGRAM         $INFORMIXDIR/etc/evidence.sh

     

    LTAPEDEV        /backups

    LTAPEBLK        512

    LTAPESIZE       0

     

    Snip-it of alarmprogram.sh  -- make sure top line is #!/bin/ksh

     

    BACKUPLOGS=Y

     

    ALARMADMIN=0

    ALARMPAGER=0

    ADMINEMAIL=

    PAGEREMAIL=

     

    BACKUP_CMD="ontape -a -d"

    RM="rm -f"

    ONSTATCMD="onstat"

     

     

    adamski cars: ll onconf.cars

    -rw-r--r-- 1 informix informix 90619 Feb  7 12:37 onconf.cars

    adamski cars: ll alarmprogram.sh

    -rwxr-xr-x 1 informix informix 20022 Feb  7 13:01 alarmprogram.sh

     

     

    I have made a full instance backup, bounced IDS and even rebooted server and still sometimes I get the logs to backup other times just the emails saying logs getting full.

     

    Anyone have a suggested starting point to troubleshoot as I been through all my notes from prior upgrades and nothing I did in the past to get logs backing up have seemed to help this time.

     

     

    John David Adamski

    Sr. Sysadmin/DBA

    Graceland University

    1 University Place, Lamoni, IA 50140

    adamski@graceland.edu

    641-784-5267

     



  • 2.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Thu February 08, 2024 11:06 AM

    When you upgrade, if you are using the ALARMPROGRAM to backup logs, you have to remember to either copy the old alarmprogram script or modify the newly installed one since log backups are disabled in the delivered script by default!



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 3.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Thu February 08, 2024 11:08 AM
    Oops, never mind. I see the changes you made to alarmprogram.sh. Red herring.

    Art

    Art S. Kagel, President and Principal Consultant
    ASK Database Management


    Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.








  • 4.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Thu February 08, 2024 11:08 AM

    The BACKUP_CMD you're citing doesn't look to be the default one, so which alarmprogram.sh is this in - v12's or v14's?

    In other words: is this really the command being executed?  Do you see log backups getting created in /backups?

    And regarding your logical logs, when getting those emails:  does "onstat -l" show multiple logs carry U (used) but no B (backed up) flag?  I.e. are logs not getting backed up when they should.

    The ALARMPROGRAM should be called with each logical log filling or, more precisely, with each switch to a new logical log.
    I'd start by tracing this script, e.g. by putting some "echo $(date) 'did this and that, on line x'  >> /tmp/alarmprog.trace".

    HTH,

     Andreas



    ------------------------------
    Andreas Legner
    ------------------------------



  • 5.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Thu February 08, 2024 04:05 PM

    Sorry my post formatting was a bit off, since last Outlook update been having problems with what I type and what is sent being totally different format wise. 

     

     

    The current alarmprogram.sh I have in place is the v12 version, and was working prior to the upgrade with no problems.  So no new stuff from v14.  Still not working.  I originally tried the v14 one with the BACKUPLOGS and BACLUP_CMD changed to what I need. 

     

    When it does work, I see a corresponding log in /backups and onstat -l shows the B to that log.  When it not working I see one or more logs with U but no B, as expected. 

     

    Even stranger I am not getting two emails each 15 odd minutes with the % full message.  Had to deal with a "the sky is falling" situation and didn't have any time today to look at.  Will add debug line in script to see if that points to something. 

     

    Hopefully I will have a quieter Friday to do some debugging.

     

    Thanks for the suggestions.

     

    john

     






  • 6.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Thu February 08, 2024 04:39 PM
    Hi,

    First make sure the alarmprogram is executable by informix and really gets called.

    For debugging the alarm events, put a line somewhere near the beginning like:
    echo "`date +%Y%m%d%H%M%S` $1,$2,$3,$4,$5" >> /tmp/alarm_${INFORMIXSERVER}.out

    You should see each event triggered in the log (informix user should be able to write to the file ...)
    After the case "$EVENT_SEVERITY" block, put another 
    echo ${BACKUPLOGS} ${LTAPEDEV} ${BACKUP_CMD} >> /tmp/alarm_${INFORMIXSERVER}.out

    You should get a 23 event (LOGICAL LOG FILE COMPLETED) for each log filling up.
    Add more output there for investigation, echo the full command call to the alarm_${INFORMIXSERVER}.out file.

    Redirect the error output of the BACKUP_CMD call to the same output to see what really happens.

    Are you using onbar or ontape ? 
    This event can be manually enforced in a testing environment with onmode -l (force next log).

    We have been using almost the same script for this purpose since version 11.x I think.
    So any script from 12.x should at least do something with the common events (which did not change).

    Hope this helps.


    MARCUS HAARMANN






  • 7.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Fri February 09, 2024 08:38 AM

    Thanks for your suggestions.  I had problems when we went from 11.x to 12.x and not sure if we are using the 11.x or the 12.x alarmprogram.  When I do a onmode -l the logs do get backed up.  When I do some sql the logs don't seem to get backup automatically.  Hopefully I will have time to later today to debug, this morning not looking promising. :-(

     

    John

     






  • 8.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Fri February 09, 2024 01:13 PM

    I was able to do some debugging and testing.  I ran a sql to update a large number of records to make sure I would get logs to roll to next.  And the old log got backed up and the alarmprogram show correct information from the debug.  Not sure what make the alarmprogram to act more like it should, as all I did was add a few debug lines, saved it and then ran the sql and worked as should.  Undoing the debug lines seem to not effect it and still seems to be working correctly.

     

    It looks like what happening now, is if the current active log is over some a percentage of full I start getting emails from database that the log is getting full.

     

    I don't exactly know what producing the email as I don't remember setting up any monitoring or warning.  So now a dungeon search to figure out what is producing the emails.  As the warning is nice but getting one every 15 minutes on a quiet system annoying.  Especially if for last few hours been at 80.7%.

     

    Thanks again for suggestions on the alarmprogram. 

     

    john

     






  • 9.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Fri February 09, 2024 04:42 AM
    Edited by Doug Lawry Fri February 09, 2024 05:17 AM

    Hi John.

    Firstly, you should never use alarmprogram.sh from a previous version. Always reapply your differences to the new version to ensure it's compatible and to have bug fixes. You should make a copy of the standard version, make your changes, then keep a copy of that in case it's overwritten in an upgrade:

    cd $INFORMIXDIR/etc
    cp -p alarmprogram.sh alarmprogram.sh.std
    vi alarmprogram.sh
    cp -p alarmprogram.sh alarmprogram.sh.new

    These are my recommended differences:

    $ diff alarmprogram.sh.std alarmprogram.sh
    28c28
    < BACKUPLOGS=N
    ---
    > BACKUPLOGS=Y
    30c30
    < ALARMADMIN=0
    ---
    > ALARMADMIN=3
    32c32
    < ADMINEMAIL=
    ---
    > ADMINEMAIL=informix@localhost # redirect via ~/.forward
    35c35,39
    < MAILUTILITY=/usr/bin/mail
    ---
    > MAILUTILITY="sendmail -t" # message headers in content
    >
    > exec 2>> $INFORMIXDIR/tmp/alarmprogram.$INFORMIXSERVER.log # record any errors
    >
    > cd # in case IDS started in non-writable directory which breaks this
    56c60
    < BACKUP_CMD="onbar -b -l"
    ---
    > BACKUP_CMD="ontape -a -d"

    The "cd" might fix your problem. If not, recording errors should reveal it.

    You should also always have email alerts configured which are frequently essential. The one you mentioned was definitely not from IDS.

    ------------------------------
    Doug Lawry
    Oninit Consulting
    ------------------------------



  • 10.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Mon February 12, 2024 02:27 PM

    Hi,

    cd does not fix all issues.

    If the the current working directory (cwd) for the server no longer exists (e.g. when started under automation) then the server will not be able to fork a process to run the alarmprogram! 

    I cannot remember if something appears in the online.log for this condition.

    Regards,
    David.



    ------------------------------
    David Williams
    ------------------------------



  • 11.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    Posted Fri February 09, 2024 04:51 AM
    Edited by Doug Lawry Fri February 09, 2024 05:18 AM

    Also, you can check whether IDS was started in a non-writable directory on Linux with

    lsof -a -d cwd -p PID

    where PID is the process ID of the main "oninit" process with PPID 1.

    ------------------------------
    Doug Lawry
    Oninit Consulting
    ------------------------------



  • 12.  RE: Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

    IBM Champion
    Posted Fri February 09, 2024 09:23 AM

    I am confused as to why it works sometimes.  I would start by just putting a statement at the start of alarmprogram to echo something to a file in /tmp to see if it is even being called.  Then dump out the environment, as I have seen problems before when the environment was not set properly when informix was started.  Are you starting informix differently now, e.g. with a service?



    ------------------------------
    Mike Walker
    xDB Systems, Inc
    www.xdbsystems.com
    ------------------------------