Originally posted by: SystemAdmin

On /dev/hd1 (/home) I have a directory /home/bad_dir that I can not remove. I can rename it as long as it stays on the same filesystem but I can not move it to another filesystem.

Beneath this directory there are two subdirectories, and they have two subdirectories. If I try and remove or unlink any of the individual subdirectories, or if I try and remove/unlink the top level /home/bad_dir, I get that the directory is not empty.

The directory is not part of a NFS export, is not linked to any other place on the system.

root@test 05$pwd
/home/bad_dir/05

root@test 05$ls -alp ./02
total 312
drwxrwxrwx 2 root system 151552 Sep 12 2007 ./
drwxrwxrwx 4 root system 4096 Sep 12 2007 ../

root@test 05$istat ./02
Inode 163904 on device 10/8 Directory
Protection: rwxrwxrwx
Owner: 0(root) Group: 0(system)
Link count: 2 Length 151552 bytes

Last updated: Mon Oct 29 16:13:35 EDT 2007
Last modified: Wed Sep 12 09:00:20 EDT 2007
Last accessed: Thu Mar 27 16:35:01 EDT 2008

root@test 05$find . -name 02 -type d -exec rm {} \;
rm: 0653-603 Cannot remove directory ./02.

root@test 05$rm -rf ./02
rm: 0653-611 Directory ./02 is not empty.

root@test 05$mv -f 02 /tmp
mv: 0653-611 Directory 02 is not empty.

What the heck is going on? How do I get the directory to go away? This server has been running for months and is a production server. I can not take it down to run fsck. Is it possible to edit the inode table and remove references to it? This box is AIX 5.3 and JFS2.

Thanks for the assistance.

Originally posted by: vinodn

can u try with rm -R ./02

Originally posted by: SystemAdmin

Thanks for the reply Vinodn. I get the same error with the -R

root@test 05$rm -R ./02
rm: 0653-611 Directory ./02 is not empty.

Does anyone know if I can edit the inode table directly and just remove the pointers to this directory?

Thanks.

Originally posted by: esv

is it a filesystem or inside the directory is there a link to a filesystem??

regards,
enrique.

Message was edited by: Enrique Sanchez

Originally posted by: SystemAdmin

Thanks for the reply Enrique,

It is a directory tree on /home (dev/hd1) that starts with bad_dir that can not be deleted or moved.

/ home
/ home /bad_dir

/ home /bad_dir/04
/ home /bad_dir/04/11
/ home /bad_dir/04/12

/ home /bad_dir/05
/ home /bad_dir/05/01
/ home /bad_dir/05/02

Can not remove /home /bad_dir
Can not remove any child of /home /bad_dir like /home /bad_dir/04 or /home /bad_dir/05/02

(There are not really spaces around /home, they were necessary to get the BOLD to work in this post)

Originally posted by: orphy

Have you tried fsck?
Orphy

Originally posted by: SystemAdmin

Orphy, this is a live production machine that I can not take down any time soon, and I had it drilled into my head as a junior admin to NEVER run fsck on a mounted file system. Frankly, I'm little afraid of what might happen if I did run it - is it safe to do so on an IBM JFS2 filesystem?

Originally posted by: orphy

Looking at PS1 of your original post, I thought this is a test box. Were you indeed root (id or whoami) when doing the rm?

Running fsck on a mounted FS doesn't do you much good but it wouldn't hurt either. I was assuming that you knew what I meant about fsck (umount && fsck). You could possibly have some corrupted inodes. I don't know how "live" this system is but you don't need to take down the entire server to run 'fsck /dev/hd1'. You just need to get all the users logged off and obviously any programs that run off hd1 will have to be stopped (but I hope you don't have any apps running off there!). The point is you want to umount /dev/hd1 and fsck it.

What ML/TL (oslevel -s) of 5.3 are you running? There are are few JFS2 APARs out there to fix corruptions but not sure if it's your case yet. And no, you should not think about manually fixing inodes in a production box... especially you are calling yourself "junior"...
Orphy

Originally posted by: SystemAdmin

Orphy:

I was root on the production box.
I changed the name of the server to "test" as a matter of privacy for my company.
I haven't been a "junior" admin for many years but the warnings from my System V NCR Tower850 days still linger on.
Yes, I know to unmount a file system before attempting the usual types of repairs, but this is a 24-7, computer and kicking all of the users off and stopping all program access to this filesystem is not feasible for a couple of months more.
/home is data only, I have /usr and /opt for programs.

While I am excluding this directory in the daily tar backups, I don't know if it is inducing any problems into my monthly mksysb builds that could come back to bite me if there is a hardware failure.

Thanks for the advice, I'll wait until I can do a clean unmount and hope fsck resolves the problem, otherwise I guess it is time to rebuild the filesystem.

Originally posted by: orphy

My bet! You did mention originally that you couldn't fsck.

Have you tried something like this? If not, give it a try as root and see how it goes.

1. cd /home/bad_dir/05/02
2. ls -al # manually rm anything, maybe 1 at a time
3. cd ..
4. rmdir 02
5. ls -al # do the same one-at-a-time here
6. cd ..
7. rmdir 05

If this works, try 04 and so on. If not, see if you can plan an fsck. Is this bad_dir causing issues to production or is it just bugging you? If it's the later, it might be good to just let it go for now until you could fsck...
Orphy

Originally posted by: tony.evans

We run fsck on mounted filesystems without any issues, you can't make any repairs but it can still hint at issues. Never had it make a filesystem worse than it already was, but it is of course up to you!

What user is bad_dir? What processes are owned by that user?

Sounds to me like you've got something open in that filesystem which a process has started removing but not finished yet (either due to a fault with it, or with the filesystem).

You can use the kernel debugger to find out which processes have a known inode open.

The instructions I have are for finding out why a filesystem won't unmount, but they may be applicable.

<hr />
Another way to search for the process still holding an inode on that FS
is the following:
1) start the kdb
2) vfs
--> This will show us all the mounted FS of the machine
--> Find the one which fails to unmount (I call it N here)
.
3) run
(0)> vfs N | awk '{ if ($3 > 0) print $1 " " $3 " " $4 " " $6 }' | more
After some lines of statistics for that FS you should see something
like
COUNT VFSP TYPE
2 1 F100009E2364A8B0 REG
3 1 F100009E2363A8B0 REG
4 1 F100009E2362A8B0 REG
5 1 F100009E2365A0B0 REG
...
.
The 2nd column is the count of users of that inode and the 3rd
column is the address of the gnode related to the inode
.
4) gnode <gnode_adr>
--> write down the gn_seg information and the gn_data information
.
5) scb 2 <gn_seg>
--> There you should search for the proc pointer (proc)
.
6) proc <proc>
--> There we have the process that has an inode open. Just
calculate the dec PID from the hex value, exit the kdb and kill
the process.
<hr />

You may be able to use that info to find out what, if anything, has any inodes open.

Originally posted by: SystemAdmin

Thanks Tony.

I tried lsof looking to see if anything was using the directory but nothing turned up. Thanks for the tip on using the kdb, pretty slick and I can see where this would be very useful in the future.

A run of fsck just now does turn up invalid directory entries so it looks like until I can unmount the filesystem, I will just have to deal with it.

Thanks everyone for your help, I'll mark this question as answered.

Originally posted by: SystemAdmin

Thanks to everyone for their help, I really appreciate the community.