Originally posted by: litfos

Hi,

On a pretty big Power6 based 570 Lpar (60 GB ram) I noticed in topas constant values of PAGING Faults (55014)/per second (with no action on PgspIn 0 or PgspOut 0)

$ vmstat 1 20

System Configuration: lcpu=16 mem=58624MB

kthr memory page faults cpu

---

-----------

---

------------

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa
3 0 5474111 904926 0 0 0 0 0 0 68 88319 8489 20 2 78 0
3 0 5474113 904922 0 0 0 0 0 0 124 96128 7878 24 2 74 0
5 0 5473875 905165 0 0 0 0 0 0 230 40793 2420 7 2 92 0
1 0 5474114 904920 0 0 0 0 0 0 126 92586 8780 20 2 77 0
3 0 5474124 904911 0 0 0 0 0 0 551 132376 14184 30 4 66 0
1 0 5472708 906331 0 0 0 0 0 0 209 39103 2442 6 1 93 0
0 0 5472688 906349 0 0 0 0 0 0 325 19154 1231 3 1 96 0
0 0 5472674 906362 0 0 0 0 0 0 368 27305 1619 13 1 86 0
2 0 5472701 906334 0 0 0 0 0 0 138 25551 893 11 1 88 0
1 0 5472791 906239 0 1 0 0 0 0 223 35071 3594 9 4 87 0
1 0 5473296 905729 0 0 0 0 0 0 103 55395 2276 14 2 84 0
1 0 5473260 905762 0 0 0 0 0 0 148 34041 2061 7 2 92 0
1 0 5474189 904813 0 1 0 0 0 0 1323 134295 10643 18 6 76 0
2 0 5474034 904966 0 0 0 0 0 0 1116 183282 9419 19 5 76 0
1 0 5473977 905023 0 0 0 0 0 0 1108 96648 9109 4 4 91 0
1 0 5474578 904421 0 0 0 0 0 0 974 107591 4252 13 4 83 0
3 0 5474726 904270 0 0 0 0 0 0 1912 146720 4960 16 5 78 0
3 0 5474767 904186 0 0 0 0 0 0 1527 142664 4976 7 7 87 0
1 0 5474323 904590 0 0 0 0 0 0 1570 128844 5114 6 5 89 0
0 0 5475644 903267 0 0 0 0 0 0 1462 237006 6630 17 6 77 0
vmstat -v
15007744 memory pages
13978606 lruable pages
902073 free pages
4 memory pools
2951661 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
63.0 numperm percentage
8819497 file pages
0.0 compressed percentage
0 compressed pages
63.0 numclient percentage
80.0 maxclient percentage
8819492 client pages
0 remote pageouts scheduled
175212 pending disk I/Os blocked with no pbuf
145758 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
190292 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults
vmstat shows high faults 'sy'(System calls) value.
Does it mean that vmm is not configured properly as I have these page faults?
Is there a mechanism to check which program generating such big number of system calls and what are they?
#AIX-Forum

Originally posted by: SystemAdmin

Hi,

system calls amount is normal (doesn't have any common with page faults). paging faults number is too high.

And this
175212 pending disk I/Os blocked with no pbuf
is also not good.

coulg you post vmo -a | grep lru ?

Regards,
Alexander
#AIX-Forum

Originally posted by: litfos

Hi Alexander,

There is a complete vmo -a

1. vmo -a

cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
kernel_psize = 16777216
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 1
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 80
maxfree = 1088
maxperm = 11182884
maxperm% = 80
maxpin = 12185114
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 15007744
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 4
minfree = 960
minperm = 2795720
minperm% = 20
nokilluid = 0
npskill = 129024
npsrpgmax = 1032192
npsrpgmin = 774144
npsscrubmax = 1032192
npsscrubmin = 774144
npswarn = 516096
num_spec_dataseg = 0
numpsblks = 16515072
page_steal_method = 0
pagecoloring = n/a
pinnable_frames = 12061429
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1
vmm_mpsize_support = 1
lru values are default (the ones which are on AIX 5.3 TL6)

best regards,
rafal
#AIX-Forum

Originally posted by: SystemAdmin

try the following:
vmo: maxperm%=90, maxclient%=90, lru_file_repage=0.
ioo: pv_min_pbuf=1024, numfsbufs=2048

Could you also be more specific about server load? Oracle?
#AIX-Forum

Originally posted by: litfos

There is a DBMS (Openingres) running with data on the filesystem (not a raw devices) + proprietary unix GUI application.

What made me wonder whether this does not create any performance , that on the machine with 60GBs or RAM there is a paging space area created with 60GB in size. I hope AIX copes well with such huge paging space.
lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging05 hdisk2 pagingvg 10240MB 4 yes yes lv
paging04 hdisk2 pagingvg 10240MB 4 yes yes lv
paging03 hdisk2 pagingvg 10240MB 4 yes yes lv
paging02 hdisk2 pagingvg 10240MB 4 yes yes lv
paging01 hdisk2 pagingvg 10240MB 4 yes yes lv
paging00 hdisk2 pagingvg 10240MB 4 yes yes lv
hd6 hdisk0 rootvg 3072MB 11 yes no lv

rafal
#AIX-Forum

Originally posted by: SystemAdmin

Don't worry, no problem with large paging space.

That brilliant prerequisites from enterprise software guru always make me laugh (SAP for instance requires 2xRAM or 3xRAM for paging space). While having intensive paging within even 10% from your 60G you may already consider your server dead. For I don't think that bunch of angry users will be satisfied with the fact that in spite of the fact that their response time now is 1000 times higher from what have been earlier, there's a large field to grow.

Regards,
Alexander
#AIX-Forum

Originally posted by: shargus

Take a look at this whitepaper:

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883

I know you are not running Oracle, but there's a lot of good information about memory tuning and recommendations, including paging space.
#AIX-Forum