Originally posted by: nicolasdenis

Hi,

We are facing a memory issue on several systems : System memory usage is very high

Using nmon

FileSystemCache 5.1%
Process 19.2%
System 72.4%
Free 2.6%

Paging space used : 90%

Using "perfpmr.sh -x memdetails.sh" we can see that most of the segments used are kernel heap segments

kernel heap 7945.00 Megabytes

Indeed using "svmon -S | grep 'kernel heap'"

da00d - work kernel heap m 4096 4096 0 4096
8a008 - work kernel heap m 4096 4096 1 4096
fa00f - work kernel heap m 4096 4096 0 4096
ca00c - work kernel heap m 4096 4096 0 4096
ba00b - work kernel heap m 4096 4096 0 4096
aa00a - work kernel heap m 4096 4096 0 4096
19a019 - work kernel heap m 4096 4096 0 4096
1ba01b - work kernel heap m 4096 4096 0 4096
12a192 - work kernel heap m 4096 4096 0 4096
13a193 - work kernel heap m 4096 4095 0 4096
9a009 - work kernel heap m 4096 4096 0 4096
ea00e - work kernel heap m 4096 4096 0 4096
1aa01a - work kernel heap m 4096 4096 0 4096
.....

Using "ps gv" we don't see this memory usage! we only see around 2.3 GB in RSS

Using our monitoring tool history, we noticed that, some days ago, system started using paging and paged over 8GB in one hour
During this period, number of processes increased and reached 50.000 then suddenly decreased to average load (around 1.500) but we don't know who/why

We would like to know

• Why kernel heap usage remains so high ?
• Is it possible to free some system memory manualy ? How ?
• What kind of process might use kernel heap but never free memory ? Is it possible that this is a bug of one of our applications ?

We are running AIX 6.1 TL6 with 12GM RAM and 10 GB paging space (hosted by VIO 2.2)
System is used for both development and test (Cobol compiled with Microfocus and C compiled with GCC)
When system is restarted everything goes back to normal (+/- 50% process, 25%system, 20% Filesystem and no paging)

Thanks in advance
#AIX-Forum

Originally posted by: alethad

What type of applications are you running specifically?

Do you have the system tuned appropriately according to what apps you're running?
Have you checked all your tuning parameters? vmo, schedo, svmon...etc.

Did someone make a change that would have affected the system this way? Or being a test/dev box someone could have found a way to push it to its limits. I mean a system just doesn't suddenly jump to using all the memory and paging without a reason.

Are you running 32 or 64-bit?

It looks more like the system might not be config'd correctly in the tuning parameters somewhere. But you will need to do some more digging to try to pinpoint it more to prove T/F.

And of course everything goes back to normal after any reboot since you just reset everything back to 0 so to speak.
Good luck.

You've got to continue to grow, or you're just like last night's cornbread -- stale & dry Loretta Lynn alethad
#AIX-Forum

Originally posted by: nicolasdenis

Ok found it.

When using find command on a particular cifs mount, multiple thousands of process are created and marked as Zombie

ppl 3474282 Z 0:00 <defunct>
ppl 3408744 Z 0:00 <defunct>
ppl 3605358 Z 0:00 <defunct>
ppl 3539820 Z 0:00 <defunct>
ppl 4260738 Z 0:00 <defunct>
...

System start using a lot of 'System memory' and paging space. Network load increases as well.

When find command is killed, all defunct processes disapear and network load comes back to normal but system memory is never freed !

I've found this APAR IZ92164 (http://www-01.ibm.com/support/docview.wss?uid=isg1IZ92164) and i thought it might be related but Fix is already installed on my system.

I'll open a support call with my vendor.
#AIX-Forum