Originally posted by: filippo gadotti

Hello,

I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help...

I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management...

Straight to the point, I have a sever which yields high number of page faults even though it has plenty of available memory.

This server runs a file-reading-intensive program and an oracle database. I have no serious performance problems so far, but these page faults started to worry me as we plan to stuff some more tasks into this 40GB men server.

I first went on and did my homework reading about AIX VMM (this is my first time with AIX servers ) and got a glance of it's peculiar way of paging everything files and programs and the way it uses a deamon to steal and clean whenever it runs short. Well... I come from Solaris, where short memory causes pages faults. Page faults causes scan-rate and scan-rate most-likely means paging. When I issue my vmstat and see the 'sr' column with 4 digit numbers in a production server, it feels... wrong...

Now, I'll past below my stats.. could someone give me more perspective of what I'm seeing?

System (from nmon startup):

¦                               6 - CPUs currently                   ¦ ¦                               6 - CPUs configured                  ¦ ¦                            1900 - MHz CPU clock rate               ¦ ¦                  PowerPC_POWER5 - Processor                        ¦ ¦                          64 bit - Hardware                         ¦ ¦                          64 bit - Kernel                           ¦ ¦                         Dynamic - Logical Partition                ¦ ¦                    5.3.7.1 ML07 - AIX Kernel Version               ¦

$ vmstat 10 5 r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec 3 1 6892117 4855669 0 1 0 543 3695 0 3333 26670 12425 82 5 13 0 2.30 91.9 5 0 6892111 4855614 0 0 0 439 3895 0 3245 74660 12366 77 13 10 0 2.37 94.9 3 0 6891110 4856689 0 0 0 635 4806 0 3129 44884 12170 80 6 14 0 2.29 91.4 4 0 6891517 4856185 0 0 0 504 4241 0 3208 41366 13178 80 6 14 0 2.29 91.5 3 0 6891105 4856693 0 0 0 388 2059 0 3162 27696 13502 82 5 12 0 2.33 93.1

$ vmstat -v 12582896 memory pages 11905361 lruable pages 4858699 free pages 1 memory pools 3055581 pinned pages 80.0 maxpin percentage 10.0 minperm percentage 20.0 maxperm percentage 19.9 numperm percentage 2379986 file pages 0.0 compressed percentage 0 compressed pages 19.9 numclient percentage 20.0 maxclient percentage 2379986 client pages 0 remote pageouts scheduled 5114 pending disk I/Os blocked with no pbuf 151149 paging space I/Os blocked with no psbuf 2484 filesystem I/Os blocked with no fsbuf 41094 client filesystem I/Os blocked with no fsbuf 8101 external pager filesystem I/Os blocked with no fsbuf 0 Virtualized Partition Memory Page Faults 0.00 Time resolving virtualized partition memory page faults

nmon shot (memory and paging):

¦ Memory ------------------------------------------------------------------------¦ ¦          Physical  PageSpace |        pages/sec  In     Out | FileSystemCache  ¦ ¦% Used       62.6%     40.5%  | to Paging Space   0.0    0.0 | (numperm) 18.7%  ¦ ¦% Free       37.4%     59.5%  | to File System    0.0  207.8 | Process   21.6%  ¦ ¦MB Used   30779.8MB  7405.8MB | Page Scans        0.0        | System    22.3%  ¦ ¦MB Free   18372.1MB 10898.2MB | Page Cycles       0.0        | Free      37.4%  ¦ ¦Total(MB) 49151.9MB 18304.0MB | Page Steals       0.0        |           ------ ¦ ¦                              | Page Faults    5474.0        | Total    100.0%  ¦ ¦------------------------------------------------------------ | numclient 18.7%  ¦ ¦Min/Maxperm     4651MB(  9%)  9301MB( 19%) <--% of RAM       | maxclient 18.9%  ¦ ¦Min/Maxfree     960   1088       Total Virtual   65.9GB      | User      35.7%  ¦ ¦Min/Maxpgahead    2      8    Accessed Virtual   27.0GB 41.0%  Pinned    24.3%  ¦ ¦                                                                                ¦ ¦ Paging-Space ------------------------------------------------------------------¦ ¦    Volume-Group PagingSpace-Name Type LPs  MB    Used IOpending                ¦ ¦          rootvg              hd6  LV  128  4096  60%    0    Active    Auto    ¦ ¦          rootvg         paging00  LV  126  4032  60%    0    Active    Auto    ¦ ¦          rootvg         paging01  LV  318 10176  25%    0    Active    Auto    ¦ ¦--------------------------------------------------------------------------------¦

topas shot:

Tue Dec 15 13:45:24 2009   Interval:  2         Cswitch   11920  Readch  3124.7K Syscall   29015  Writech 2256.6K Kernel    9.4   |###                         |  Reads       703  Rawin         1 User     82.4   |########################    |  Writes      326  Ttyout      238 Wait      0.0   |                            |  Forks         3  Igets         0 Idle      8.2   |###                         |  Execs         3  Namei      2381 Physc =  2.40                     %Entc=  96.1  Runqueue    4.5  Dirblk        0 Waitqueue   0.0 Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out en4    6243.7   5640.5  1053.0  6179.9    63.7  PAGING           MEMORY lo0       0.0      0.0     0.0     0.0     0.0  Faults     3226  Real,MB   49151 Steals        0  % Comp     42.2 Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PgspIn        0  % Noncomp  18.8 hdisk3    2.0     1.7K   28.5     0.0     1.7K  PgspOut       0  % Client   18.8 hdisk23   1.5   512.9     4.0     0.0   512.9   PageIn        3 hdisk14   0.5   172.3    35.1     0.0   172.3   PageOut     560  PAGING SPACE hdisk2    0.0     0.0     0.0     0.0     0.0   Sios        563  Size,MB   18304 % Used     40.0 Name            PID  CPU%  PgSp Owner           NFS (calls/sec)  % Free     60.0 java        1208454  68.5 144.4 util            ServerV2       0 syncd        348408   4.8   0.5 root            ClientV2       0   Press: java        1364098   1.1  60.0 root            ServerV3       0   
"h" 

for help topas        577692   0.0   2.0 util            ClientV3       0   
"q" to quit

I appreciate any feedback!

cheers!

f.
#AIX-Forum

Originally posted by: alethad

I believe the first thing you need to do is read over the Whitepaper "Oracle Tuning on AIX 5L"
This will explain a lot to you and give you the tuning parameters you need to change to tune your Oracle a little better with your OS. I gave this to someone else in this forum the other day.

I will attach the original and the latest version of the whitepaper since you didn't state which OS or Oracle versions you are running.
I will have to do them separate since this forum only allows 1 attachment per post.
Read over these then get back to us.

Originally posted by: alethad

Here is the original version.
Keep these on hand you will need them again I'm sure. They are updated also as AIX and Oracle release new versions so you will want to check on them now and again.

Originally posted by: filippo gadotti

Right, thanks for the docs!

I was reading through it and found some recommendations regarding VMM tunning which are far away from what I currently have. i.e. maxperm is recommended to be at 90% while mine is only 20%

I've been also studing some further and it seems that the system is alright, since even with these whole lot of faults there isn't a lot of I/O happening:

$ vmstat -I 2   System configuration: lcpu=6 mem=49151MB ent=2.50   kthr     memory              page              faults              cpu -------- ----------- ------------------------ ------------ ----------------------- r  b  p   avm   fre  fi  fo  pi  po  fr   sr  in   sy  cs us sy id wa    pc    ec 5  0  0 11640858 130272   3 176   0   0   0    0 1969 14780 8209 78 18  4  0  2.49  99.4 6  0  0 11640981 129758   3 216   0   0   0    0 2118 13378 6155 74 23  3  0  2.49  99.7 6  1  0 11640952 129454   0 863   0   0   0    0 2372 13326 4965 73 24  3  0  2.48  99.1 5  0  0 11641176 128860  12 133   0   0   0    0 1906 54047 6640 73 24  3  0  2.48  99.1 4  0  0 11640962 128620  11 208   0   0   0    0 2657 54259 8458 77 19  4  0  2.49  99.5 5  0  0 11639816 129358   9 191   0   0   0    0 2450 49697 5415 81 17  2  0  2.50  99.8 7  0  0 11639772 128988   8 144   0   0   0    0 2501 52366 6225 79 18  3  0  2.50  99.9

Actually, what I have in mind is that these page faults are simply being generated by my poor VMM settings:

80.0 maxpin percentage 10.0 minperm percentage 20.0 maxperm percentage 19.9 numperm percentage

Still, I wish I could tell preciselly the reason of the page faults.

thanks!

f.
#AIX-Forum

Originally posted by: alethad

Unless you understand the internals of the application and the OS you may not be able to do that.

Once you get the system tuned properly get with your DBA for help in monitoring how the application is now performing. Sometimes what appears to be an issue on the system side may not be anything to really worry about or a problem that affects the application. Sort of like a false positive.

Good luck.
#AIX-Forum

Originally posted by: filippo gadotti

Hey!

Thanks man!
Well to be honest I can't really touch the config flags. I'm more of application/dev level, so I'm trying to figure out what I could tell from my pov what could be done.
For now, I can only report what I've found to be 'normal' according to the current settings, and forward the white book as a guide line.
Besides, these flags seem to have been changed already. I guess (hope) the guys doing it had reason and knowledge for that.

Now I have to say that I find a bit unsaddling this of not being able to figure out the page faults... I'll find frustrating that for fine tuning the memory management I have to become an specialist in every program in the server. Now, I do agree that at some level of specificity (and Oracle often gets there) knowledge of the app helps. but.. I wish the world was more like JVM's GC, in which you can watch the generations going and figure you flags out.

That said, I'll keep on the white papers and forums :)

thanks!

f.
#AIX-Forum

Originally posted by: alethad

Oracle really shouldn't be paging that much if it's configured correctly. Or at least 9i and 10g shouldn't. These pretty much run in Memory unless you're a little short in that area. You can set AIX to force Oracle not to page. I forget how to do that but it can be done. There might be a reference about it in the original doc I sent you.

Now remember there are parameters on the Oracle side of the fence that you DBA can also set to fine tune the Oracle server with the OS.

Well good luck in your quest.
#AIX-Forum

Originally posted by: SystemAdmin

Have a look at your maxclient setting (vmo -o maxclient%). If this is set to 20 (same as maxperm%) then that is why you were seeing the high scan rate to free ratio. Maxclient% specifies the max percentage of RAM that can be used to cache client pages. The numclent figure is 19.9, which indicates it may well be 20 and is therefore at this level. Assuming that is the case the page stealer comes along and ensures that the number of client pages is lower than 20% of memory, even if there is more memory available and it is having to do a lot of scanning in order to find some pages it can free up.

So I agree with the other posts. You have no memory or paging issues as such - you just need some tuning of your vmo parameters (and possibly your paging space allocation). Read the docs but FYI our primary Oracle systems are tuned with:

lru_file_repage=0
lru_poll_interval=10
maxclient%=90
maxperm%=90
minperm%=5

Hope this helps,

Sam
#AIX-Forum

Originally posted by: Montecarlo

Hi
A page fault occurs whenever a process requests data that does not already exist in memory. According to the statistics you posted, you have 37% of memory free. Paging rates to paging space are 0. There are some page outs to file system space - this is normally a result of data read, modified and written back to disk.
The scan rate is "Pages scanned by page-replacement algorithm."

As far as AIX is concerned, memory is not stressed. You have three paging spaces, all are different sizes. AIX recommends that all paging spaces are the same size and located on separate disks. You have some paging space in use - the topas report indicates that java processes are using paging space - this is not unusual.

Other than changing the sizes of the paging spaces there is no indication that memory requires attention.
Regards, Simon
#AIX-Forum