Originally posted by: jetyvand

Hello friends,

The attached graph shows paging statistics fetched from nmon on one of my servers for a few days. Unlike the other servers that I have, this one have huge scans and stable high faults. What Im really interested in is what type of usual behavior that provoke huge scans similar to this.. and what scans data from nmon really is...
#AIX-Forum

Originally posted by: SystemAdmin

Hi,

When the number of free memory pages drops below minfree, the lrud runs to release pages until maxfree is reached.
Maybe you have some workload that uses a lot of memory on this server, like backups or batchs...

Have a look to AIX Virtual Memory management method to better understand what's happening.
#AIX-Forum

Originally posted by: UNIX-OS

during scan rate spike you may be running backup or batch jobs.

page fault may be running out of file system cache, how much memory you have on this server, can you post vmstat -sv output and vmstat -wtI 2 5 command output?

Thanks
#AIX-Forum

Originally posted by: jetyvand

Yes, I have backupjobs on this server. But unlike the other servers, this one have these huge spikes.. and paging faults.

output:

root::/tmp> vmstat -wtl 2 5 System configuration: lcpu=8 mem=18432MB ent=1.20 kthr          memory                         page                       faults                 cpu              large-page       time ------- --------------------- ------------------------------------ ------------------ ----------------------- ----------------- -------- r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa    pc    ec      alp      flp hr mi se 2   0    4106138       4009     0     0     0     0      0     0   152 309802 40660 46 24 30  0  1.43 119.2        0        0 09:52:41 2   1    4110508       2084     0     0     0  1235   1235     0   260 166677 55816 55 20 23  1  1.70 142.0        0        0 09:52:43 2   1    4104955       7986     0     0     0   181    181     0   324 175069 75726 57 19 21  2  1.73 144.0        0        0 09:52:45 3   1    4107307       5614     0     0     0     0      0     0   178 139033 44071 54 19 25  3  1.57 130.9        0        0 09:52:47 2   0    4103380      25289     0     0     0  7884  22967     0    87 147256 30926 41 24 34  1  1.08  90.3        0        0 09:52:49

root::/tmp> vmstat -sv 15994027152 total address trans. faults 1562190397 page ins 357801690 page outs 0 paging space page ins 0 paging space page outs 0 total reclaims 11001642518 zero filled pages faults 262848 executable filled pages faults 4526626932 pages examined by clock 10 revolutions of the clock hand 1714902861 pages freed by the clock 56819195 backtracks 1764087 free frame waits 0 extend XPT waits 48512463 pending I/O waits 1915802762 start I/Os 135044728 iodones 65037794376 cpu context switches 308190269 device interrupts 3571093287 software interrupts 2206619594 decrementer interrupts 330303 mpc-sent interrupts 330303 mpc-receive interrupts 205936931 phantom interrupts 0 traps 252291048482 syscalls 4718592 memory pages 4564672 lruable pages 6182 free pages 1 memory pools 1004377 pinned pages 80.0 maxpin percentage 5.0 minperm percentage 20.0 maxperm percentage 7.0 numperm percentage 319565 file pages 0.0 compressed percentage 0 compressed pages 7.0 numclient percentage 20.0 maxclient percentage 319565 client pages 0 remote pageouts scheduled 18 pending disk I/Os blocked with no pbuf 0 paging space I/Os blocked with no psbuf 1972 filesystem I/Os blocked with no fsbuf 0 client filesystem I/Os blocked with no fsbuf 24760 external pager filesystem I/Os blocked with no fsbuf 93.1 percentage of memory used 

for computational pages

#AIX-Forum

Originally posted by: SystemAdmin

UNIX-OS wrote:
The output needed was vmstat -Iwt (capital i)
You are using a lot of computational pages, maybe programs running are memory consuming or you have a memory leek.
Your VMM parameters (maxclient,maxperm) prevents using more than 20% of whole memory for file caching.

Can you send output of these commands :

svmon -P -O summary=basic,unit=MB
svmon -G -O unit=MB
ps -elfk | sort -rnk 10 | head -10

PS : I saw that the same post is in Performance Forum, maybe you should keep only one...
#AIX-Forum

Originally posted by: UNIX-OS

From first vmstat I don't see any issue, only few threads waiting in blocked status, CPU wait is in single digit. Only last line I see its scanning 3 time more pages to free up 7884 page, this is not too bad. System is not doing lot of IO during this time.
capture vmstat -wtI 2 command output during peak load time for couple of hours and see scan and free page ratio.

From second vmstat output, revolutions of the clock hand is 10, how long this server is up?

maxperm percentage set to 20%, Is this oracle database server?
What kind of work load runs on this server?
What is the value for strict_maxclient? What is the OS version?
As meyawi said server is using more computational memory. Not enough space the file system cache. That may be the reason your seeing more page fault.

Thanks
#AIX-Forum

Originally posted by: jetyvand

Thank you for your help so far! I will post more detailed information when I come back to work Monday.
#AIX-Forum