Originally posted by: SystemAdmin

Hi

I have a confusion over the memory usage in AIX. The senerio is:
We earlier had an AIX box which has 32GB of memory supporting 1500 users and Oracle database of 4GB. There I use to see 70-75% of memory usage(nnmon) i.e around 23-24GB of memory.We migrated to a new one with 128GB memory and users increased to somewhere around 2100 and of course oracle database to 6GB. Oracle is using about 24GB and 16GB by one more application(as from svmon -U).Apart from these there is no significant application running.But the memory usage now showing is 99%(nmon). Why is this so? It seems the increase in memory utilization is not proportionate to the increase in load.Also we are dealing with 128GB of memory this time.I am worried because if there is a sharp increase in load the system will start paging. I am scared of memory leaks.

Our server use "Deferred page space allocation" policy and maxperm=80, minperm=20, minfree=960. At present numperm=65%.I understand that AIX loves to keep the "file pages" in memory even if the processes exits, while freeing up those in paging space.Here i have a confusion. What happens to these file pages eventually? It seems if these pages are not used for a long time they will be pushed to the paging space by the page stealer. If so,what happens to these pages if they are not used at all even after that? If not how AIX gets rid of these pages?

Please advice me how to find and get rid of these file caches.What do I need to bring down the memory usage?

with regards
kuntal


#AIX-Forum

Originally posted by: SystemAdmin

Kuntal: I'm not sure why you only had 70-75% memory usage before but the behaviour you're seeing now is what you should expect and isn't in any way a problem. You don't need to do anything about the file cache pages. Operating systems exist to handle things like this for you. Long unreferenced file pages will eventually be "reclaimed" by AIX if the contents haven't changed, this being just a fancy word for thrown away. File pages are only paged out if the contents have been changed and the owning process is still active. For the most part this should apply only to a tiny subset of buffer cache pages.

Why do you think this is a problem?

Regards,
Jim Lane
#AIX-Forum

Originally posted by: SystemAdmin

Hi,

Thanks for the kind reply. Ok that seems to be a normal behaviour of AIX. But I am worried for how it will behave/takecare if we have a increase in load which may demand more memory.At present though memory usage is 99% actual memory requirement is less, as I believe there is some page caching going on. This will give raise to paging to the disk to accomodate the new load. Here we will have unnecessary paging even though, in reality we have not utilize the memory and it is been used for caching !!

How do I find out the actual usage of memory by the applications and what is the amount of caching of any form not associated with the present processes running on the server?

With regards
Kuntal
#AIX-Forum

Originally posted by: SystemAdmin

In general, DBMSs will use their own filesystem caching mechanism which is preferable over AIX filesystem cache.

Rule of thumb: small FS-cache for DB-Servers, larger for Application Servers with File IO.

In the past, minperm%, maxperm% and maxclient% (for jfs2) were used to control filesystem cache sizes.

In recent issues of AIX, there was a tunable added named lru_file_repage to control the aix algorithm for file system caching.

Setting its value to 0 and minperm% to low and maxperm%/maxclient% to a high value is good advice for many purposes, since it ensures that as long as the aix filesystem cache is above minperm%, computational (process) memory pages are preferred to file pages when pages for deletion from real memory are examined.

There is a good article on developerworks: "Tuning your memory settings, Part 1"

hth
Joachim Gann
#AIX-Forum

Originally posted by: SystemAdmin

Hi Joa

Thanks. That helped me a lot.

with regards
Kuntal
#AIX-Forum

Originally posted by: SystemAdmin

Kuntal: if you add extra workload to a box heavily over-configured with RAM (I suspect that's where you are now) AIX will gradually reallocate memory, giving more and more to process working sets and less and less to the file cache. As this is going on you will observe exactly nothing. The system will continue on happily not paging and running just fine. Eventually you will "hit the wall" when you reach the point where AIX can't comfortably give out any more working set memory. You will know this has happened when you start to see paging space paging and, probably, degraded application performance. Whether you need to worry a lot about this depends on how much more work you expect to add to the box, and how unreasonable your management is likely to get when presented with unwelcome surprises. If you do want to worry then you need to look at a tool called svmon which can measure memory use by process. It is a thing sufficiently complicated as to ensure that you will be "living in interesting times".

FWIW
Jim Lane
#AIX-Forum

Originally posted by: SystemAdmin

Hi

Jim:Thanks for the valuable inputs. At present we dont have any plan to increase the load.Now I know exactly how AIX handles memory. Management was surprised to see 99% memory utilization and I ws not able to explain that.Now I am confident and able to explain them. I checked the history of the server and did not see any paging activity. This is exactly according to that you have explained. At present we dont have any performance problem.Thanks again.

With regards
Kuntal
#AIX-Forum

Originally posted by: SystemAdmin

Hi

I still have a peculiar problem with this server. We have two filesystems being used for dumping oracle backups and generally consists of big files. When these filesystems are mounted the memory utilization reaches 99% and system starts paging. On unmounting these file systems memory utilization comes down and paging stops. Why is this happening? Mine is a database server running oracle rack and the following is my vmo settings (below);

memory = 128GB
total memory usage from #svmon -U | grep -ip User is around 42GB.

1. svmon -G

size inuse free pin virtual
memory 32768000 32731018 36982 5325340 12547328
pg space 32768000 2843065

work pers clnt
pin 5325071 0 269
in use 11984359 0 20746659

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 32291770 2836569 4986492 12105248
m 64 KB - 27453 406 21178 27630

1. vmo -a

cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
kernel_psize = 16777216
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 1
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 80
maxfree = 1088
maxperm = 25260364
maxperm% = 80
maxpin = 26432554
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 32768000
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 8
minfree = 960
minperm = 6315088
minperm% = 20
nokilluid = 0
npskill = 256000
npsrpgmax = 2048000
npsrpgmin = 1536000
npsscrubmax = 2048000
npsscrubmin = 1536000
npswarn = 1024000
num_spec_dataseg = 0
numpsblks = 32768000
page_steal_method = 0
pagecoloring = n/a
pinnable_frames = 27445234
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1
vmm_mpsize_support = 1
Not sure how to handle this. As a workaround I am unmounting the two offending filesystems once the backup is done.

With regards
Kuntal
#AIX-Forum

Originally posted by: hdkutz

Follow the instructions on
http://www.ibm.com/developerworks/aix/library/au-aixoptimization-memtun1
There are a lot of useful hints to memory tuning.

Cheers,
ku
#AIX-Forum