Originally posted by: apple08

Dear IBM AIX gurus,
would like to seek your advice. I was wondering our /usr keep 100%. i have increased the size, it dropped and in a day it will grow to 100%. how to locate or identify the files that is very dynamic and cause the increase. would it possibly of other thing? i tried to use this command.
find /usr -ctime +1 -name "*.log" -exec ls -l {} \; |more
but it list file more than a year ago, but the command was find in another aix server. is something wrong somewhere. how do u normally do with this kind of strange?
hope to hear from you. thank you

Originally posted by: tony.evans

Try

find /usr -xdev -mtime -1 -ls

also, man find is helpful.

Originally posted by: apple08

dear Sir,
what this output mean?

1. find /usr -xdev -mtime -1 -ls |more

2 8 drwxrwxrwt 16 bin bin 8192 Jun 3 17:30 /usr

im sorry, im in learning stage. hv a nice day

Originally posted by: SystemAdmin

This means that in the last day one or more files were created or modified in /usr and deleted after, because they are not listed in the find output.

Regards,
Andres Cordoba.

Originally posted by: flodstrom

Well, not much to go at really, but you seem to have some odd and rather dangerous permissions set on the /usr

Do a full listing of both / and /usr (ls -al) and look for files and/or directories that should not be there, especially those not owned by root or bin.

Other than that, do you have many users on the system? Are you sure you don't have univited "guests" (hacked?)?

Originally posted by: tony.evans

Well the permissions on /usr do look a little scary, you should consider how they got that way.

There is a less sinister reason for a filesystem continuing to fill up despite find not locating the cause - files which were open and removed, and are still in use. For example, if an application is logging output to a file and the file is open, and someone removes the file with rm (to do some housekeeping), the application will continue writing to the file handle it had, and AIX will continue sticking data into the filesystem, but you won't be able to see where it's going.

This can explain the difference between a df showing a 100% full filesystem and a du -sm showing insufficient files to fill a filesystem.

You could use fuser (or lsoff) to see if any processes still believe they're writing to /usr, you could check /usr with fsck in read only mode and see if it believes there's some corruption or inode confusion.

You'll need to learn how to read man pages as well. man find will tell you how to use find to locate files changed in the last few days just to make sure. Try,

find /usr -xdev -mtime -5 -ls

Originally posted by: apple08

dear Sir,
thank you
i don't understand why the permission is dangerous.would really appreciate your advice.
ls -la |grep usr
drwxr-xr-x 45 root system 4096 May 30 10:48 usr

Sir,
one of our websphere application log is written to /usr, can we do symbolic link so that the log will go to different directory other than /usr. we have tried to tell them to point their log to different directory, but they said it requires more change of coding etc.

/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs

what is the best way? hope to hear from you. many thanks in advance. thank you

Originally posted by: flodstrom

Did you change the permissions on /usr ? The reason for asking is that from your previous post it looked like your /usr was "wide open" for anyone (writable for any user), but in the latest post the permissions on /usr looks normal? Does anyone but you have root access to the system?

When you increased the size of /usr, how much did you increase it by?

Is it possible to provide more details? Like output of "df -k", "du -sk /usr/*"?

Originally posted by: apple08

dear Sir,
ohh i see.
only system admins have the root password.
i have been increasing by 6GB now become 14GB. im at home, will provide the output tomorrow. don't hv access to the server from home.
Sir, how about the symbolic link for the log in /usr

how to safely do it? let say we want to link the log in /usr to /disc1

1. cd /usr/IBM/WebSphere/AppServer/profiles/AppSrv01
2. ls

logs

1. ln -s logs /disc1

would really appreciate your advice. hv a nice day.

Originally posted by: flodstrom

Well, I don't know much about WebSphere so I can't advice on moving the log directory elsewhere.

However, if it indeed are the log files filling up /usr then you should perhaps consider moving them to a larger file system. First I would try to reduce the amount of logging it does and look into rotating the logs, etc.

In case you do want to move the log directory, just make sure that WebSphere is not running while you move the log directory and set-up the symbolic link.

Originally posted by: alethad

The link works pretty well. I've got my logs linked out to another FS because the WebSphere product I have has certain pieces of it is hard coded for what WebSphere wants to do. But with the link you get to choose where you want the logs to live. It also seems to be fine when updating.

But you should still look at turning down the amount of logging going on as suggested.

Originally posted by: tony.evans

As a personal preference, I'd create a new filesystem and mount it over logs/ rather than using links. You still need the app stopped and you need to move any logs you want to keep out before mounting the new filesystem.

I find it less confusing than having soft links pointing directories into other filesystems and allows for finer control.

Originally posted by: apple08

dear Sir,
it become 100% again. wondering since the %Iused was not up and down, just now was 85%, now drop to 74%, but the utilisation still 100%.

1. df -g |grep usr

/dev/hd2 16.00 0.15 100% 130766 74% /usr

1. du -sk /usr/*

0 /usr/HTTPServer
12395676 /usr/IBM
135576 /usr/IBMIHS
40 /usr/TT_DB
374824 /usr/WAS_FIXES
0 /usr/X11R6
0 /usr/adm
292 /usr/aix
832 /usr/artic960
34756 /usr/bin
33048 /usr/ccs
0 /usr/dict
0 /usr/doc
80660 /usr/dt
0 /usr/emgrdata
76040 /usr/es
4 /usr/etc
10908 /usr/icu4c
10760 /usr/include
76844 /usr/java14
516 /usr/lbin
242612 /usr/lib
0 /usr/linux
311396 /usr/local
0 /usr/lost+found
0 /usr/lpd
1941928 /usr/lpp
0 /usr/man
83164 /usr/mozilla
0 /usr/netscape
20 /usr/openssh
202084 /usr/opt
3704 /usr/pegasus
320 /usr/perfagent
3240 /usr/pmapi
0 /usr/pub
3924 /usr/samples
175816 /usr/sbin
9932 /usr/share
6012 /usr/sni
0 /usr/spool
0 /usr/src
2408 /usr/ssa
312 /usr/suma
2892 /usr/swlag
100 /usr/sys
140 /usr/sysv
0 /usr/tmp
40 /usr/ucb
0 /usr/usg
317292 /usr/websm

hope to hear from you. many thanks in advance.

Originally posted by: apple08

dear Sir,
would like to add, i have run this command, attached is the output for your comments.
find /usr -xdev -mtime -1 -ls |more

hope to hear from you. thank you

Originally posted by: apple08

to avoid anything during the weekends, i have increased by 2GB. now the usr size is 18GB. i feel its very big.

Originally posted by: tony.evans

18GB is far too big for usr, for a start you'll struggle to do mksysb's anywhere useful.

You need to,

1. stop your application
2. tidy up any log files
3. turn down logging
4. clean up any core dumps
5. create filesystems / soft-links so that your application is no longer using /usr for logging (my preference is filesystems in non-rootvg volume groups)
6. restart your application

There's no quick fix simple answer, you need to put in the work.

Originally posted by: apple08

dear Sir,
thank you. will do.
after zoom in using the find command, so many heapdump files created by websphere. The websphere created quite a numbers of heapdump files which only been created when websphere crashed.
our application will look into it. after housekeep all the websphere dump files,the utilisation has dropped from 100% to 44% as shown below and we have 10GB free space, once everything is stabilised and fine, do u recommend us to reclaim back the space. smit jfs2 and reduce the size. Sir, the inode used is highest but the inode percentage is only 6% as compared to the rest of the filesystems, do we need to bother this inode?:

1. df -g |grep usr

/dev/hd2 18.00 10.13 44% 130747 6% /usr

many thanks in advance. thank you for all your support. hv a nice day.