Hello Again,
I modified the cronjob to start at 20h local time. I let it run, and it finished without errors; file system filled up to about 73% after which the backup script deleted the oldest backup and we were back to 50%. So it is definitely not the file system filling up, despite the errno 28.
I have searched the system logs; maybe the process was killed for whatever reason: nothing found. This file system is only used for the ontape backup, nothing else is writing to it.
I am going to write some monitoring scripts and let them run during the backup, see what gives.
Arjen
------------------------------
Arjen Van Drie
------------------------------
Original Message:
Sent: Wed July 14, 2021 11:44 AM
From: Arjen Van Drie
Subject: ontape does not finish the backup: tape full
Ok,
let me do just that, even though I can hardly imagine the file system filling up, it actually only contains the informix ontape backup. But I have been surprised before.
Funny enough it happened today, yesterday, July 6th, 5th, June 30th, 29th, 28th, 21st.. On the other days I got a "Completed" in the logs.
I will run a manual backup now and see what is going on...
Thanks so far,
Arjen.
------------------------------
Arjen Van Drie
Original Message:
Sent: Wed July 14, 2021 11:21 AM
From: Art Kagel
Subject: ontape does not finish the backup: tape full
Arjen:
I have to agree with Paul, errno 28 is:
$ pr_err 28
Error number: 28 Message: No space left on device
The filesystem was full at the moment that ontape declared it so. Did you look?
It's possible that while ontape does not write out an uncompressed file and hand that to the BACKUP_FILTER, it may be that depending on the filesystem you are using that the file is not being overwritten immediately. Run a 'df -k' on the filesystem to see if it has space before you kill the ontape session next time it fails. If it does, then I would use du to find out what's eating the file system space. Start at the top of the filesystem's mount point, run:
du -s -k *|sort -n
and drill down into the largest offending directories until you figure it out.
Art
------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
Original Message:
Sent: Wed July 14, 2021 10:20 AM
From: Arjen Van Drie
Subject: ontape does not finish the backup: tape full
Hello Paul,
I can hardly imagine that would be the problem: the backup filesystem is 1TB in size of which half is used. The expected size of the backup is about 260G. I assume that ontape sort-of pipes the backup through the BACKUP_FILTER program before writing to disk, and not afterwards (first writing the uncompressed backup, then compressing it)?
------------------------------
Arjen Van Drie
Original Message:
Sent: Wed July 14, 2021 09:46 AM
From: Paul Watson
Subject: ontape does not finish the backup: tape full
28 is out of space - are you filling the FS
Paul Watson
Oninit LLC
+1-913-387-7529
www.oninit.com
Oninit®️ is a registered trademark of Oninit LLC
Original Message:
Sent: 7/14/2021 9:39:00 AM
From: Arjen Van Drie
Subject: ontape does not finish the backup: tape full
Hi all,
We have a cron job running at 22:00 hrs to backup our database. I have no clue why ontape is aborting. I hope I have entered enough info below. The resulting bzip compressed file is 240G one day, 253G the next, so it is aborting on different sizes and also after different runtimes. I just found out: it is happening irregularly. On some days it completes, on other days it aborts. OS is CentOS Linux 7.9. Has someone seen this type of behavior?
I have tried to restore on a different server, and ontape -r actually asks for tape 2...
Further info:
$] grep ^TAPE onconfig.ol_rbpc
TAPEDEV /opt/backup/informix/full/ontape.back.bz2
TAPEBLK 32
TAPESIZE 0
$] grep FILTER onconfig.ol_rbpc | grep -v ^#
BACKUP_FILTER /bin/lbzip2
RESTORE_FILTER /bin/lbunzip2
ol_rbpc.log:
22:00:02 Level 0 Archive started on rootdbs, physdbs, logdbs, datadbs, ol_rbpc, sbspace
[...]
23:38:00 Archive on rootdbs, physdbs, logdbs, datadbs, ol_rbpc, sbspace ABORTED.
23:38:00 Aborted by client.
/tmp/ontape.log: (a redirect of STDOUT and STDERR)
Please enter the level of archive to be performed (0, 1, or 2)
Please mount tape 1 on /opt/backup/informix/full/ontape.back.bz2 and press Return to continue ... Using the backup and restore filter /bin/lbzip2.
Read/Write End Of Medium enabled: blocks = 8260876
Tape is full ...
Please label this tape as number 1 in the arc tape sequence.
Please mount tape 2 on /opt/backup/informix/full/ontape.back.bz2 and press Return to continue ... function put arc data failed code -1 errno 28
Using the backup and restore filter /bin/lbzip2.
10 percent done.
20 percent done.
30 percent done.
40 percent done.
50 percent done.
60 percent done.
70 percent done.
80 percent done.
Interrupt received ...
Best regards,
Arjen.
------------------------------
Arjen Van Drie
------------------------------
#Informix