Originally posted by: SystemAdmin

Hi,

We've fenced a number of our AIX database servers in the past with no issues.

When trying to fence two AIX 5.3 servers via:

minperm%=5
maxclient%=25;
maxperm%=25%

nmon shows the Min/Maxperm correctly set at 5/25
but FileSystemCache (numperm) is still 80.0%
and memory free is still: % Free 0.0%

The VMO parameters were changed after a reboot of the servers with all
Oracle databases stopped.

Any idea what is causing the vmo memory fencing to fail?

Below is a list of the active vmo parameter settings.
--nmon-v10r---h=Help-------------Host=nhcbp55a-------Refresh=2 secs---22:12.28--
-Memory-Use---------------------Paging------------------------Stats-----------

Physical PagingSpace pages/sec In Out FileSystemCache

% Used 100.0% 0.3% to Paging Space 0.0 0.0 (numperm) 80.0%

% Free 0.0% 99.7% to File System 2338.5 76.2 Process 14.5%

MB Used 30715.9MB 29.2MB Page Scans 18085.2 System 5.5%

MB Free 4.1MB 8418.8MB Page Cycles 0.0 Free 0.0%

Total(MB) 30720.0MB 8448.0MB Page Steals 2259.3 ------

Page Faults 1323.8 Total 100.0%

Min/Maxperm 1470MB( 5%) 7348MB( 24%) note: % of memory

Min/Maxfree 960 1088 Total Virtual 38.2GB User 44.1%

Min/Maxpgahead 2 8 Accessed Virtual 5.9GB 15.3% Pinned 5.9%

------------------------------------------------------------------------------

cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 1
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 25
maxfree = 1088
maxperm = 1880996
maxperm% = 25
maxpin = 6358837
maxpin% = 80
mbuf_heap_psize = 4096
memory_affinity = 1
memory_frames = 7864320
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 0
minfree = 960
minperm = 376199
minperm% = 5
nokilluid = 0
npskill = 16896
npsrpgmax = 135168
npsrpgmin = 101376
npsscrubmax = 135168
npsscrubmin = 101376
npswarn = 67584
num_spec_dataseg = 0
numpsblks = 2162688
page_steal_method = 0
pagecoloring = n/a
pinnable_frames = 7381820
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1


#AIX-Forum

Originally posted by: nagger

The min and max numbers you are setting are thresholds used in the memory allocation algorythms. They effect a) the choosing of which type of memory to free when memory is in short supply and b) when daemons run.

They do not immediately FORCE memory to be reallocated as that may cause a very large performance hit.

As your numperm is still high 80% - I assume you have run for some time after setting min and max perm - this highlights that AIX can nothing better to use this memory than filesystem cache and that there is not pressure on memory i.e. it is not being demanded. The numperm will only come down if thee are lots of requests for memory from process for process memory or things like shared memory segments and AIX will then prefer to free up file system cache pages due to numperm being higher than maxperm and hence numperm reduced.

At the moment numperm is 80-25=55% high than you expected i.e. 55% more memory is full of filesystem pages and this will lead to much higher cache hit ratios for filesystem disk I/O which is a very good thing. What would be the point of flushing out all these pages and having 55% of your memory unused? Absolutely, none what so ever. If you start an application that does deman memory then AIX will quickly reassign the filesystem cache memory to process memory for you. this is the best of both worlds.

If you have a filesystem based database, for example, this will speed it up by avoiding disk I/O - although you might consider making the RDBMS caches larger so data is cached better within the DB.

This is all well documented in the AIX manuals and Redbooks.

If you insist on wasting RAM consider use of the tuning options:
strict_maxclient - you have this on
strict_maxperm - you have this off

But unless you run an application that demands 555 of memory strict_maxperm=1 will slow your machine down in 95% of cases.

I hope this helps, N
#AIX-Forum

Originally posted by: SystemAdmin

Thanks for your reply Nigel.

The thing is that fencing AIX Oracle database memory is not as simple an issue,
as your reply seems to imply.

I'm referencing the "IBM Oracle Technical Brief - Oracle Architecture and Performance Tuning on AIX" by Damir Rublic November 1, 2005 pp 22 recommendations.

What happens on AIX Oracle database servers with a min/max default setting of
20/80 is that AIX keeps stealing application code pages, which result in continuous paging of application code.

If an Oracle backup or log archive runs during the day, AIX will try to use 80%
of the memory as OS file buffers, causing heavy paging of the Oracle database.

Since Oracle has its own database buffer cache allocated within its SGA,
its better to fence the OS file buffer space at a minimum value (10-25%),
and leave free memory for Oracle user thread connections.

When memory is fenced in this fashion, backups and archives only use up to the
10-25% OS file buffers memory threshold and do not impact an Oracle instance.

This has been an issue with AIX Oracle database servers for some time.

And the impact is not just when the backup ran.
IBM Tivoli TSM seems to hold on to that memory long after the backup is finished. Since a TSM process doesn't terminate after a backup is completed,
its working pages (OS file buffers) don't appear to return immediately to the free list. It appears they are added back to the free list when they are stolen by the VMM for other programs.

Response time on the Oracle database can be impacted for hours.

Fencing AIX memory at a 5/10 or 5/25 min/max threshold provides:
1. Protection from paging
2. Identifiable free memory for DBA Oracle allocation
3. Protection from IBM Tivoli TSM

#AIX-Forum

Originally posted by: dannert

Hi,

as Nigel already pointed out setting strict_maxperm to 1 would enforce your maxperm settings.

I agree with Nigel that that is typically not optimal as memory is not fully utilized. I would try to leave strict_maxperm at 0 and instead set, via vmo,lru_file_repage=0.

If the value of the lru_file_repage parameter is set to 0 and the number of file pages exceeds the value of the minperm parameter, file pages are selected for replacement. If the number of file pages is lower than the value of the minperm parameter, any page that has not been referenced can be selected for replacement.

This means as long as your minperm parameter is set correctly so that physical memory is available to back up that many file pages and you are not overcommited on memory from the application side, AIX will only utilize file pages to satisfy new memory requests and not computational pages ==> you should see basically no paging to paging space.
#AIX-Forum

Originally posted by: SystemAdmin

Thanks Dannert.

I'll try out the options you mentioned.

I have to disagree with you on "typically not optimal as memory is not fully utilized". After a certain point there's little or no performance gain
allowing file buffer memory to grow from 1-2GB to 20-24GB on an Oracle database server.

Oracle has it's own database buffer area as part of it's SGA.
If a database instance is getting 85-95% hit ratio's within its database buffer
cache, of around 1-2GB, having the OS file buffers grow to 80% of memory,
or 20-24GB on a 30GB server, doesn't provide much aof a benefit.
It's actualy a waste of memory and of dollars spent.

And what happens, a DBA looks at NMON on the server, and see's free memory
at 0%,and orders more memory before adding additional Oracle instances.

If you search the web, there appears to be a lot of DBA's running into this
issue, and they are all trying to address the issue by using
vmo maxperm/minperm options.

Thanks again,

Joe

#AIX-Forum

Originally posted by: SystemAdmin

We have very similar issues with our Oracle and Redbrick database servers. For Oracle, we have lots SGA pages that end up on paging space.

I just found this document today:
http://www-941.ibm.com/collaboration/wiki/download/attachments/436/VMM+Tuning+Tip+-+Proctecting+Comp+Memory.pdf?version=1

by Barry Saad at ATS.

It states that the practice of setting: maxperm%=maxclient% to a low value (say your 25%), minperm%=5, strict_maxperm=0, strict_maxclient=0 is a "legacy" approach.

Instead the "new" method is to set maxperm% to a high value, set the new parameter lru_file_repage=0 and leave strict_maxclient=1 (default) (plus the other details in the pdf).

Until I found this document, I was inclined to go with the "legacy" method. Any thoughts on what to do?????
#AIX-Forum

Originally posted by: VirtualGreg

Use the latest recommendations. Also, if warranted, you can pin your SGA with the correct Oracle and vmo options. Somewhere around, there is a document that talks to using a lock_sga parameter (or something like that) in Oralce and shmpin in vmo.
#AIX-Forum

Originally posted by: SystemAdmin

Thanks. That was my plan.

I had a little bit of reservation about doing this as I had read a document from IBM/Oracle joint testing center (can't find the link now) that recommended against pinning the SGA because it would preserve the dynamicity of the SGA -- the SGA can grow to the max size, but with v_pinshm=1 and SGA_LOCK=TRUE in init.ora, the SGA would never be able to shrink (unless you restart the instance of Oracle). This wouldn't be a problem at my site. However, I'd still like to follow best practices.

Yet this document doesn't proscribe pinning shared memory:

http://www-03.ibm.com/servers/enable/site/peducation/wp/9a46/9a46.pdf#search=%22ibm%20oracle%20SGA%20v_pinshm%22
#AIX-Forum

Originally posted by: niella

Also consider the following (from Bruce Spencer's website), regarding the new tuning method:
"Update 6/20/06: A customer reported this approach can cause excessively high paging when there are multiple, unbalanced memory pools. In this case, IBM Support suggested disabling memory affinity. If you run into this behavior, I recommend calling IBM Support and reference this tip."

I've only had performance issues like this on one of our servers, multiple other work fine.

Regards,
Niel
#AIX-Forum