Informix

Informix

Connect with Db2, Informix, Netezza, open source, and other data experts to gain value from your data, share insights, and solve problems.

 View Only
Back to discussions
Expand all | Collapse all

ontape does not finish the backup: tape full

  • 1.  ontape does not finish the backup: tape full

    Posted 07/14/21 09:39 AM
    Hi all,

    We have a cron job running at 22:00 hrs to backup our database. I have no clue why ontape is aborting. I hope I have entered enough info below. The resulting bzip compressed file is 240G one day, 253G the next, so it is aborting on different sizes and also after different runtimes. I just found out: it is happening irregularly. On some days it completes, on other days it aborts. OS is CentOS Linux 7.9. Has someone seen this type of behavior?

    I have tried to restore on a different server, and ontape -r actually asks for tape 2...

    Further info:

    $] grep ^TAPE onconfig.ol_rbpc   
    TAPEDEV /opt/backup/informix/full/ontape.back.bz2
    TAPEBLK 32
    TAPESIZE 0

    $] grep FILTER onconfig.ol_rbpc | grep -v ^#
    BACKUP_FILTER /bin/lbzip2
    RESTORE_FILTER /bin/lbunzip2

    ol_rbpc.log:
    22:00:02  Level 0 Archive started on rootdbs, physdbs, logdbs, datadbs, ol_rbpc, sbspace
    [...]
    23:38:00  Archive on rootdbs, physdbs, logdbs, datadbs, ol_rbpc, sbspace ABORTED.
    23:38:00  Aborted by client.

    /tmp/ontape.log: (a redirect of STDOUT and STDERR)                                      
    Please enter the level of archive to be performed (0, 1, or 2)  
    Please mount tape 1 on /opt/backup/informix/full/ontape.back.bz2 and press Return to continue ... Using the backup and restore filter /bin/lbzip2.
    Read/Write End Of Medium enabled: blocks = 8260876  

    Tape is full ...  

    Please label this tape as number 1 in the arc tape sequence.  

    Please mount tape 2 on /opt/backup/informix/full/ontape.back.bz2 and press Return to continue ... function put arc data failed code -1 errno 28
    Using the backup and restore filter /bin/lbzip2.
    10 percent done.
    20 percent done.
    30 percent done.
    40 percent done.
    50 percent done.
    60 percent done.
    70 percent done.
    80 percent done.

    Interrupt received ...


    Best regards,
    Arjen.

    ------------------------------
    Arjen Van Drie
    ------------------------------

    #Informix


  • 2.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 09:47 AM
    28 is out of space - are you filling the FS

    Paul Watson
    Oninit LLC
    +1-913-387-7529
    www.oninit.com
    Oninit®️ is a registered trademark of Oninit LLC





  • 3.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 10:20 AM
    Hello Paul,

    I can hardly imagine that would be the problem: the backup filesystem is 1TB in size of which half is used. The expected size of the backup is about 260G. I assume that ontape sort-of pipes the backup through the BACKUP_FILTER program before writing to disk, and not afterwards (first writing the uncompressed backup, then compressing it)?


    ------------------------------
    Arjen Van Drie
    ------------------------------

    Original Message


  • 4.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 10:49 AM

    TAPESIZE in the onconfig set to zero?  Backups to disk running into a tape full and asking for a second tape should not happen.

     

    Hal Maner

    M Systems International, Inc.

     




    Original Message


  • 5.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 11:05 AM

    Hal:

    As Paul said, you will get a tape full message if the filesystem on which the TAPEDEV file or directory is located fills up. It looks like sometimes there is enough space for the archive file and other times there is not enough space. That may be because the file is sometimes bigger or because there are other data being written to that file system periodically and later deleted.

    Art



    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 6.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 11:21 AM
    Arjen:

    I have to agree with Paul, errno 28 is:

    $ pr_err 28
    Error number: 28  Message: No space left on device

    The filesystem was full at the moment that ontape declared it so. Did you look?

    It's possible that while ontape does not write out an uncompressed file and hand that to the BACKUP_FILTER, it may be that depending on the filesystem you are using that the file is not being overwritten immediately. Run a 'df -k' on the filesystem to see if it has space before you kill the ontape session next time it fails. If it does, then I would use du to find out what's eating the file system space. Start at the top of the filesystem's mount point, run:
    du -s -k *|sort -n

    and drill down into the largest offending directories until you figure it out.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 7.  RE: ontape does not finish the backup: tape full

    Posted 07/14/21 11:44 AM

    Ok,

    let me do just that, even though I can hardly imagine the file system filling up, it actually only contains the informix ontape backup. But I have been surprised before.

    Funny enough it happened today, yesterday, July 6th, 5th, June 30th, 29th, 28th, 21st.. On the other days I got a "Completed" in the logs.

    I will run a manual backup now and see what is going on...

    Thanks so far,
    Arjen.



    ------------------------------
    Arjen Van Drie
    ------------------------------

    Original Message


  • 8.  RE: ontape does not finish the backup: tape full

    Posted 07/15/21 07:11 AM
    Hello Again,

    I modified the cronjob to start at 20h local time. I let it run, and it finished without errors; file system filled up to about 73% after which the backup script deleted the oldest backup and we were back to 50%. So it is definitely not the file system filling up, despite the errno 28.

    I have searched the system logs; maybe the process was killed for whatever reason: nothing found. This file system is only used for the ontape backup, nothing else is writing to it.

    I am going to write some monitoring scripts and let them run during the backup, see what gives.

    Arjen

    ------------------------------
    Arjen Van Drie
    ------------------------------

    Original Message


  • 9.  RE: ontape does not finish the backup: tape full

    Posted 07/15/21 07:08 AM


    Hi Arjen,

    When the backup completes successfully does it prompt for a second tape?

    If the ontape prompts for a 2nd tape and the prompts are not responded to correctly then the behavior is unclear when there is a backup filter.

    You could be getting a errno 28 when the filesystem is not full especially as there a backup filter - ontape is writing to a pipe not the filesystem.

    ENOSPC The device containing the file referred to by fd has no
    room for the data.

    The fd (file descriptor here) is the pipe to the backup filter not the filesystem

    Also what is TAPESIZE set to?

    Regards,

    David.



    ------------------------------
    David Williams
    ------------------------------



  • 10.  RE: ontape does not finish the backup: tape full

    Posted 07/15/21 07:20 AM
    Edited by System Admin 01/20/23 04:28 PM
    Hi David,

    no, on success it does not prompt for another tape. I have restored yesterday's backup on another machine and all went well.

    So the problem would be in lbzip2 (multi threaded bzip2). That could indeed be the problem when piping data into it as opposed to a single threaded compression utility. Problem is that the uncompressed level 0 backup is too big for the file system, and using a single threaded bzip2 takes too long for it to finish.

    Is using a backup filter _exactly_ the same as writing to STDOUT and then piping to the compression utility as far as you know?

    Best,
    Arjen.

    ------------------------------
    Arjen Van Drie
    ------------------------------

    Original Message


  • 11.  RE: ontape does not finish the backup: tape full

    Posted 07/15/21 08:45 AM
    Arjen:

    No, using a BACKUP_FILTER is not the same as writing to STDOUT and piping to a compression utility. When it uses the BACKUP_FILTER ontape/onbar writes the header information block to the file first uncompressed then pipes the rest of the archive to the filter and directs the filter to append the compressed data to the file.

    Keep in mind that you are using a multi-threaded compressor, it has to coordinate writing the compressed data blocks to the output from the multiple threads so that they land in the output file in the correct order. I don't know exactly how lbzip2 handles this, but it is possible that temporary storage is involved and that might sometimes suck up the remaining storage when both the old and new archive files are a bit larger. Your test was during a successful run, it tells us nothing about what's happening when the run fails.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------

    Original Message


  • 12.  RE: ontape does not finish the backup: tape full

    Posted 07/15/21 01:02 PM
    True that is, but I don't think that lbzip2 uses a different algorithm each time it runs. So when the problem would be in the compression utility, it would fail each time, I'd say.

    Anyway, I will look and test further and if all fails I probably need to acquire new hardware (bigger, better, probably SSD), so I can write the backup uncompressed.

    Thanks again,
    Arjen.

    ------------------------------
    Arjen Van Drie
    ------------------------------

    Original Message


  • 13.  RE: ontape does not finish the backup: tape full

    Posted 07/19/21 12:23 PM


    Hi,

    I would consider changing BACKUP_FILTER (and RESTORE_FILTER) to a shell script which calls the compression program but appends stderr from that to a fixed file so you can see any error messages from the compression program.

    All,

    Where does the stderr of the BACKUP_FILTER/RESTORE_FILTER go by default? stderr of the oninit process?

    Regards,

    David.



    ------------------------------
    David Williams
    ------------------------------

    Original Message


Related Content