Originally posted by: ragh

We are running weblogic product tests on AIX boxes. While running tests at some point test is going to hung mode and after that if user type any command its giving error "fork: Not enough space". Its clear that issue is with memory and we use to hard reboot the machine to resolve this problem. With respect to this problem I have couple of questions.

1. At this type of situation even root user can't execute any commands at console, is there any way we can set some limit to user process,? at least it should allow root user to run the command.
Note: when I run "ulimit -Ha" command ( on two servers I am getting below o/p for "max user processes")

max user processes (-u) 262144 ---( On one server)
max user processes (-u) 128 --- ( On another server)
So I am not getting meaning of these two because our test goes to hung mode on both server approx at same point.

Originally posted by: SystemAdmin

Hi ragh,

First of all check your Paging Space using lsps -a, it should be nearly 1.5 to 2 times of your physical memory (RAM). Increase Paging Space by smitty chps if required.

We get a fork error when memory usage is exceeded, try and reduce the traffic/requests on client side (if any) moving to server and the server will let you login or do activities after some time without rebooting.

Ya....seems that you need to set your limits to "unlimited" status. Set all options under default column as -1 under /etc/security/limits file. And also for weblogic app user (if any).

Originally posted by: ragh

Thanks Superman,
But I am facing one more problem with respect to performance. we are running series of tests... ( Note:we are using NIS account to Login into the box). At some point of time our test will struck, we usually kill those and will start next series of tests. ( After waiting 15 - 20 mins). But each time I am noticing AIX box is not freeing up the memory and paging space it was utilizing before it keeps on adding for next category of tests. I am not understanding which process is holding memory.

I have attached screen shot of topas o/p before we start last category of tests. If you see even though there is no big process, memory %comp is 85% and it is utilizing some paging space also.

My query is

a.) Is there any we can find exactly which process is holding how much memory or paging space.

Originally posted by: CRM

Just to confirm one point below first, hopefully one of the memory tuning experts can confirm.

You say AIX is not freeing memory, not true as far as I was aware, AIX is the memory manager but it is the APPLICATION that requests and returns memory to AIX. If you have removed all ulimits so that applications can use as much memory as they can get hold of, it will be a badly tuned application that is eating up all of the memory.

I have seen something like this before and the easiest way I found to monitor this was to write a very simple svmon based shell script that just ran an svmon once a minute and piped this to a file.

The box will need to be rebooted once memory fills up and you get the fork error but the ouput of svmon will be there in the text file and you can see which application was using all of the memory.

regards

Chris