Originally posted by: KrisB
I know this is an old post but it came up in a search for me this weekend. I had a similar problem and didn't find much either. My oracle database stopped being able to write to the archive log directory and reported the following errors.
Sun Oct 18 05:47:36 2009
Errors in file /u02/oracle/admin/prod/bdump/prod1/prod1_arc0_807124.trc:
ORA-19502: write error on file "/archprod1/arch/prod1/prod_redo1_79236_622224263.arc", blockno 4097 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 28: No space left on device
Additional information: -1
Additional information: 1048576
Checking the space on the filesystem we had plenty of space. We pointed the archive logs to another directory, I unmounted the filesystem and ran fsck.
The current volume is: /dev/prodarch01lv
File system is currently mounted.
Primary superblock is valid.
fsck: Performing read-only processing does not produce dependable results.
-
Phase 1 - Initial inode scan
Inode 4240 has bad size (NOT FIXED)
Cannot repair inode 4240 (NOT RELEASED)
Cannot repair inode 4397 (NOT RELEASED)
-
Phase 2 - Process remaining directories
-
Phase 3 - Process remaining files
-
Phase 4 - Check inode allocation map
File system inode map is corrupt (NOT FIXED)
-
Phase 5 - Check block allocation map
Block allocation map is corrupt (NOT FIXED)
Block allocation map is corrupt
File system is currently mounted.
fsck: Performing read-only processing does not produce dependable results.
I made a backup copy of the existing logs in this fileystem and then ran fsck and let it fix errors.
The current volume is: /dev/prodarch01lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/prodarch01lv
Primary superblock is valid.
-
Phase 1 - Initial inode scan
Inode 4240 has bad size; FIX? y
Cannot repair inode 4397; RELEASE? y
Superblock marked dirty because repairs are about to be written.
-
Phase 2 - Process remaining directories
Inode 4397 is linked as: /arch/prod1/prod_redo1_79216_622224263.arc
Directory inode 4128 has an invalid reference to inode 4397 in entry prod_redo1_79216_622224263.arc; REMOVE? y
-
Phase 3 - Process remaining files
-
Phase 4 - Check and repair inode allocation map
File system inode map is corrupt; FIX? y
-
Phase 5 - Check and repair block allocation map
Block allocation map is corrupt; FIX? y
File system is clean.
Superblock is marked dirty; FIX? y
All observed inconsistencies have been repaired.
ran fsck again....
The current volume is: /dev/prodarch01lv
Primary superblock is valid.
J2_LOGREDO:log redo processing for /dev/prodarch01lv
Primary superblock is valid.
-
Phase 1 - Initial inode scan
-
Phase 2 - Process remaining directories
-
Phase 3 - Process remaining files
-
Phase 4 - Check and repair inode allocation map
-
Phase 5 - Check and repair block allocation map
File system is clean.
I tried copying back some of the old files to see if I continued to get errors in the errpt. All seemed fine so we pointed the logs back to this filesystem and so far no more issues.
Logged a call with support today to try to find out what could have caused this but support said since it's fixed they have no way of knowing what could have caused it. They say it could be 1 of 3 things:
1. heavy I/O
2. filesystem corruption
3. loss of communication between node and storage
errpt didn't have anything to suggest #3 and my SVC and SAN reported no problems to the effect. About 3hrs earlier we had upgraded to TL9 SP4 using multibos my gut tells me something there may have caused corruption but since I have no snap so I can't be sure. Anyway posting this in case it helps someone else in the future as this gentleman's post helped me.
#AIX-Forum