Originally posted by: AIX_KGB
Hi,
We have AIX7.1 TL3 SP1 running, after some time we experience a lot of memory problems (fork failed etc etc).
We noticed that the pinned-memory reaches 98%, but the paging-space is not used (approx. 2%). We configured "maxpin" to 90%, but that doesn't seem to work. Is there any configuration item so that de paging-space is used? Or is there any garbage-collection done at memory? One LPAR is a NIM-server, on this server we can trigger this behaviour with a "nim -o check <lpp_source>" command, every time this command is executed...the pinned-memory increases with a few %. When the command is finished the pinned-memory is not decreased. Note that we don't think it is related to the nim-command, because other (no NIM-server) LPAR's have the same problem.
Do we have memory-leakage?, is there a way how we can find the problem.
Any hints, tips or tricks would be appriciated.
Below some environment information:
┌─topas_nmon──W=WLM────────────Host=<lparclient>────Refresh=2 secs───10:08.55─────
│ Memory ──────────────────────────────────────────────────────────────────────────
│ Physical PageSpace | pages/sec In Out | FileSystemCache
│% Used 98.9% 2.1% | to Paging Space 18.0 0.0 | (numperm) 1.3%
│% Free 1.1% 97.9% | to File System 4.5 0.0 | Process 33.4%
│MB Used 1518.6MB 168.8MB | Page Scans 0.0 | System 64.1%
│MB Free 17.4MB 8023.2MB | Page Cycles 0.0 | Free 1.1%
│Total(MB) 1536.0MB 8192.0MB | Page Steals 0.0 | ------
│ | Page Faults 32.5 | Total 100.0%
│------------------------------------------------------------ | numclient 1.3%
│Min/Maxperm 40MB( 3%) 1186MB( 90%) <--% of RAM | maxclient 90.0%
│Min/Maxfree 960 1088 Total Virtual 9.5GB | User 22.2%
│Min/Maxpgahead 2 8 Accessed Virtual 1.6GB 17.2%| Pinned 93.4%
│ | lruable pages 337232.0
# vmo -a
ame_cpus_per_pool = n/a
ame_maxfree_mem = n/a
ame_min_ucpool_size = n/a
ame_minfree_mem = n/a
ams_loan_policy = n/a
enhanced_affinity_affin_time = 1
enhanced_affinity_vmpool_limit = 10
esid_allocator = 1
force_relalias_lite = 0
kernel_heap_psize = 65536
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
maxfree = 1088
maxperm = 303508
maxpin = 358788
maxpin% = 90
memory_frames = 393216
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
minfree = 960
minperm = 10116
minperm% = 3
msem_nlocks = 0
nokilluid = 12
npskill = 16384
npswarn = 65536
num_locks_per_semid = 1
numpsblks = 2097152
pinnable_frames = 59728
relalias_percentage = 0
scrub = 0
thrpgio_inval = 1024
thrpgio_npages = 1024
v_pinshm = 0
vm_mmap_bmap = 1
vmm_default_pspa = 0
vmm_klock_mode = 2
wlm_memlimit_nonpg = 1
# vmstat -v
393216 memory pages
337232 lruable pages
4615 free pages
2 memory pools
367785 pinned pages
90.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
1.1 numperm percentage
3786 file pages
0.0 compressed percentage
0 compressed pages
1.1 numclient percentage
90.0 maxclient percentage
3786 client pages
0 remote pageouts scheduled
1845 pending disk I/Os blocked with no pbuf
25706 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
1451 client filesystem I/Os blocked with no fsbuf
58 external pager filesystem I/Os blocked with no fsbuf
97.9 percentage of memory used for computational pages
# lsps -s
Total Paging Space Percent Used
8192MB 1%
# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 8192MB 1 yes yes lv 0
# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd6 paging 256 256 1 open/syncd N/A
# vmo -o defps
defps = 1
# swap -l
device maj,min total free
/dev/hd6 10, 2 8192MB 8023MB
# svmon -G
size inuse free pin virtual mmode
memory 393216 387531 5685 366824 425920 Ded
pg space 2097152 43465
work pers clnt other
pin 317062 0 962 48800
in use 381166 0 6365
PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 136363 20137 126104 145680
m 64 KB - 15698 1458 15045 17515
L 16 MB - 0 0 0 0
S 16 GB - 0 0 0 0
# vmo -L
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
ame_cpus_per_pool n/a 8 8 1 1K processors B
--------------------------------------------------------------------------------
ame_maxfree_mem n/a 24M 24M 320K 1536M bytes D
ame_minfree_mem
--------------------------------------------------------------------------------
ame_min_ucpool_size n/a 0 0 5 95 % memory D
--------------------------------------------------------------------------------
ame_minfree_mem n/a 8M 8M 64K 1535M bytes D
ame_maxfree_mem
--------------------------------------------------------------------------------
ams_loan_policy n/a 1 1 0 2 numeric D
--------------------------------------------------------------------------------
enhanced_affinity_affin_time
1 1 1 0 100 numeric D
--------------------------------------------------------------------------------
enhanced_affinity_vmpool_limit
10 10 10 -1 100 numeric D
--------------------------------------------------------------------------------
esid_allocator 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
kernel_heap_psize 64K 0 0 0 16M bytes B
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 8E-1 D
lgpg_size
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 16M bytes D
lgpg_regions
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
maxfree 1088 1088 1088 16 314572 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
maxperm 303508 303508 S
--------------------------------------------------------------------------------
maxpin 358788 358788 S
--------------------------------------------------------------------------------
maxpin% 90 90 90 1 100 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
memory_frames 384K 384K 4KB pages S
--------------------------------------------------------------------------------
memplace_data 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_mapped_file 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_shm_anonymous 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_shm_named 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_stack 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_text 2 2 2 0 2 D
--------------------------------------------------------------------------------
memplace_unmapped_file 2 2 2 0 2 D
--------------------------------------------------------------------------------
minfree 960 960 960 8 314572 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm 10116 10116 S
--------------------------------------------------------------------------------
minperm% 3 3 3 1 100 % memory D
--------------------------------------------------------------------------------
msem_nlocks 0 0 0 0 1M numeric B
--------------------------------------------------------------------------------
nokilluid 12 0 12 0 4G-1 uid D
--------------------------------------------------------------------------------
npskill 16K 16K 16K 1 2M-1 4KB pages D
--------------------------------------------------------------------------------
npswarn 64K 64K 64K 1 2M-1 4KB pages D
--------------------------------------------------------------------------------
num_locks_per_semid 1 1 1 1 64 numeric B
--------------------------------------------------------------------------------
numpsblks 2M 2M 4KB blocks S
--------------------------------------------------------------------------------
pinnable_frames 25412 25412 4KB pages S
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
scrub 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
thrpgio_inval 1K 1K 1K 0 64K numeric D
--------------------------------------------------------------------------------
thrpgio_npages 1K 1K 1K 0 64K numeric D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
vm_mmap_bmap 1 1 1 0 2 numeric D
--------------------------------------------------------------------------------
vmm_default_pspa 0 0 0 -1 100 numeric D
--------------------------------------------------------------------------------
vmm_klock_mode 2 -1 -1 -1 3 numeric B
--------------------------------------------------------------------------------
wlm_memlimit_nonpg 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
n/a means parameter not supported by the current platform or kernel
Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented
d = deprecated: deprecated and cannot be changed
Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60
Some vmstat output information before, during, after nim-command:
# vmstat 1 20
System configuration: lcpu=2 mem=1536MB ent=0.10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 0 418549 4177 0 0 0 0 0 0 35 50743 2718 49 50 1 0 0.99 990.8
11 0 418591 4276 0 0 8 8 8 0 203 49152 2698 48 51 1 0 0.98 983.9
3 0 418601 4265 0 0 0 0 0 0 26 51858 2694 49 51 1 0 0.98 982.9
1 0 418555 4454 0 0 9 9 9 0 34 52131 2734 49 51 1 0 0.99 994.1
3 0 418806 4202 0 0 0 0 0 0 22 51600 2718 49 50 1 0 0.98 983.7
3 0 418851 4155 0 0 0 0 0 0 28 51473 2676 49 50 1 0 0.99 994.3
2 0 418874 4274 0 0 9 9 9 0 28 51298 2725 49 50 1 0 0.99 990.4
4 0 418949 4193 0 0 0 0 0 0 35 50609 2630 50 50 1 0 0.99 986.0
2 0 419038 4067 0 0 0 0 0 0 38 50354 2685 50 50 1 0 0.99 986.7
5 0 419051 4224 0 74 44 251 919 0 140 50491 2859 49 50 1 0 0.99 987.5
3 0 419137 4106 0 0 0 257 381 0 41 50043 2707 50 50 1 0 0.99 986.5
3 0 419195 4169 0 0 9 9 11 0 44 49954 2678 49 50 1 0 0.99 986.0
3 0 419257 4104 0 0 0 0 0 0 27 50170 2698 49 50 1 0 0.99 989.6
3 0 419049 4592 0 0 15 18 19 0 42 51496 2729 48 52 1 0 0.99 987.4
4 0 419428 4197 0 7 0 0 0 0 32 50919 2714 49 50 1 0 0.99 988.7
2 0 419412 4213 0 0 0 0 0 0 25 50535 2678 49 50 1 0 0.99 990.6
2 0 419501 4085 0 0 0 0 0 0 33 49845 2628 49 50 1 0 0.99 987.0
2 0 419568 4011 0 0 0 0 0 0 28 50403 2681 49 50 1 0 0.98 983.4
9 0 419628 4092 0 0 5 8 10 0 31 50513 2647 49 50 1 0 0.99 988.2
5 0 419635 4063 0 1 0 0 0 0 45 50065 2698 49 50 1 0 0.99 987.0
# vmstat 1 20
System configuration: lcpu=2 mem=1536MB ent=0.10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
5 0 419993 4098 0 2 1 8 8 0 34 51789 2775 48 52 1 0 0.98 983.9
4 0 420052 4019 0 0 0 0 0 0 27 51028 2666 49 50 1 0 0.99 994.1
4 0 420068 4258 0 0 0 254 686 0 36 50715 2725 49 50 1 0 0.98 984.3
5 0 420176 4266 0 0 0 257 2035 0 41 50697 2699 49 50 1 0 0.98 984.8
5 0 420225 4339 0 2 9 9 9 0 40 50718 2713 50 50 1 0 0.99 986.8
3 0 420300 4253 0 0 0 0 0 0 30 50819 2688 49 50 1 0 0.99 988.3
3 0 420362 4214 0 0 8 137 138 0 99 49026 2816 49 50 1 0 0.97 968.4
0 0 420389 4181 0 0 0 0 0 0 27 50582 2623 49 50 1 0 0.99 987.9
3 0 420366 4337 0 0 9 9 10 0 40 50740 2651 48 51 1 0 0.99 986.7
3 0 420535 4166 0 0 0 0 0 0 20 50841 2742 49 51 1 0 0.99 990.6
0 0 420609 4226 0 0 9 9 11 0 37 50225 2668 49 50 1 0 0.99 989.6
5 0 420662 4544 0 0 0 785 1208 0 41 49414 2704 49 50 1 0 0.99 986.4
1 0 420755 4590 0 0 8 8 9 0 38 50872 2698 49 50 1 0 0.99 988.2
3 0 420819 4516 0 0 0 0 0 0 28 51180 2612 49 50 1 0 0.99 987.4
3 0 420915 4553 0 0 8 9 11 0 44 51854 2742 50 50 1 0 0.99 987.7
3 0 420960 4496 0 0 0 0 0 0 30 50945 2752 50 50 1 0 0.99 987.5
3 0 420998 4536 0 0 9 9 9 0 44 49664 2626 49 50 1 0 0.99 986.6
2 0 421141 4391 0 0 0 0 0 0 26 51112 2678 50 50 1 0 0.99 987.7
1 0 420967 4552 0 0 0 0 0 0 31 52076 2779 48 51 1 0 0.99 985.4
2 0 421250 4284 0 0 8 8 8 0 40 51078 2810 49 50 1 0 0.99 990.3
# vmstat 1 20
System configuration: lcpu=2 mem=1536MB ent=0.10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
12 0 421896 4113 0 142 38 3995 6302 0 747 32877 3232 40 59 1 0 0.80 800.5
1 0 422086 4045 0 19 0 2931 4419 0 598 30036 2783 42 56 1 0 0.81 805.6
2 0 422332 3955 0 15 10 2964 4590 0 524 29487 2713 41 57 1 0 0.81 810.3
3 0 422260 4235 0 5 68 2478 4029 1 567 29669 2857 41 58 1 0 0.81 805.9
4 0 422294 4112 0 2 7 2332 3346 1 557 29355 2772 41 57 1 0 0.81 811.8
3 0 422357 4126 0 0 416 3140 3734 0 554 28927 2765 41 58 1 0 0.81 814.0
1 0 422400 4191 0 0 447 3356 5475 0 556 29370 2723 41 58 1 0 0.82 820.9
1 0 422519 4032 0 7 0 2999 4402 0 475 27272 2604 45 53 1 0 0.83 830.4
2 1 422639 4058 0 14 1 2860 4427 0 503 27095 2741 43 55 1 0 0.82 823.9
3 0 422800 4047 0 0 9 2611 3905 0 466 27094 2555 42 56 1 0 0.82 821.3
2 1 422879 4138 0 1 54 2723 4872 0 439 25117 2513 45 53 1 0 0.82 819.8
3 0 423079 3999 0 17 7 2352 3610 0 494 26779 2518 42 57 1 0 0.83 829.9
5 0 423076 4108 0 19 0 2306 3064 0 480 26518 2499 42 56 1 0 0.84 837.4
2 1 423372 3968 0 63 110 2478 4781 0 517 25724 2618 43 56 1 0 0.83 829.2
3 0 423287 4130 0 311 0 2431 4410 0 760 26559 3156 43 55 1 1 0.84 837.2
2 0 423493 3960 0 5 0 1628 2566 0 453 25964 2441 43 56 1 0 0.83 832.6
3 0 423406 4194 0 0 8 1590 2913 0 448 25965 2444 44 55 1 0 0.84 841.9
2 0 423501 4104 0 44 0 801 1278 0 476 26080 2515 44 54 1 0 0.85 848.0
4 0 423599 3995 0 0 0 601 906 0 413 26050 2426 44 55 1 0 0.85 848.0
3 0 423625 4192 0 2 8 1021 2153 0 449 25623 2521 44 54 1 0 0.85 849.1
# vmstat 1 20
System configuration: lcpu=2 mem=1536MB ent=0.10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 0 423778 3994 0 17 2 1865 2140 0 441 25002 2480 44 55 1 0 0.84 842.5
0 0 423907 4231 0 3 11 1269 2574 0 400 24874 2426 44 55 1 0 0.85 849.4
1 0 424054 3944 0 58 133 283 570 0 494 25140 2697 45 54 1 0 0.85 850.2
1 0 424055 3969 0 2 97 1207 1983 0 402 23947 2314 46 53 1 0 0.85 854.2
0 0 424093 3948 0 1 73 1926 3308 0 422 23606 2413 43 55 1 0 0.85 852.1
1 0 424090 4274 0 0 8 2581 3623 0 398 23253 2377 44 55 1 0 0.85 846.6
3 0 424238 4271 0 6 13 1798 3033 0 410 23324 2342 45 54 1 0 0.86 864.0
2 0 424332 4110 0 102 148 1869 3178 1 523 22911 2510 45 54 1 0 0.86 857.8
0 0 424477 4127 0 55 0 2019 2027 0 469 23861 2480 44 55 1 0 0.87 865.2
2 0 424503 4036 0 0 0 744 1009 0 346 23661 2173 45 54 1 0 0.86 864.4
1 0 423671 4198 0 352 9 1154 2508 0 853 24280 3220 50 48 1 1 0.75 748.7
2 0 423990 4004 0 33 0 692 1284 0 199 37930 2572 63 37 1 0 0.96 962.2
4 0 423353 4479 0 27 0 1017 1739 0 606 130903 2689 38 61 1 1 0.87 874.
4 0 423479 4203 0 20 9 267 350 0 242 49213 3009 49 50 1 0 0.95 951.5
2 0 423511 4293 0 126 13 256 1075 0 171 49859 2864 49 50 1 0 0.98 981.2
5 0 423564 4239 0 0 0 0 0 0 38 50945 2665 49 50 1 0 0.99 990.4
6 0 423622 4299 0 12 9 9 11 0 64 49732 2633 49 50 1 0 0.99 987.0
5 0 423676 4229 0 0 0 0 0 0 31 50552 2754 49 50 1 0 0.99 988.4
4 0 423608 4409 0 3 3 9 31 0 36 51667 2754 48 52 1 0 0.99 989.3
2 0 423786 4231 0 0 0 0 0 0 25 50616 2623 49 50 1 0 0.99 991.3
# vmstat 1 20
System configuration: lcpu=2 mem=1536MB ent=0.10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
7 0 423979 4154 0 300 8 523 1430 0 360 50923 3229 49 50 1 0 0.99 992.9
2 0 424002 4111 0 0 0 0 0 0 30 50460 2696 50 50 1 0 0.99 987.6
3 0 424074 4282 0 36 18 18 18 0 80 49863 2807 50 50 1 0 0.99 986.2
4 0 424137 4186 0 29 0 0 0 0 58 49899 2751 50 50 1 0 0.99 988.6
1 0 424201 4340 0 33 0 255 422 0 72 49949 2685 49 50 1 0 0.99 987.7
3 0 424238 4300 0 0 0 0 0 0 37 50565 2664 49 50 1 0 0.99 986.0
1 0 424275 4201 0 1 0 0 0 0 53 51160 2628 49 50 1 0 0.98 982.6
0 0 424356 4257 0 0 7 8 29 0 35 51307 2678 48 51 1 0 0.98 983.6
3 0 424411 4201 0 0 0 0 0 0 25 50729 2677 49 50 1 0 0.99 989.1
2 0 424466 4293 0 0 9 138 1549 0 41 49008 2677 49 50 1 0 0.98 980.3
3 0 424562 4180 0 1 0 0 0 0 279 49636 2731 48 51 1 0 0.99 985.8
5 0 424630 4445 0 0 0 776 1320 0 40 50069 2643 50 50 1 0 0.99 988.6
5 0 424667 4521 0 24 9 9 9 0 58 50992 2779 49 50 1 0 0.99 988.6
6 0 424735 4453 0 0 0 0 0 0 24 51452 2691 49 50 1 0 0.99 988.9
5 0 424794 4516 0 0 3 9 10 0 33 51009 2692 49 50 1 0 0.99 987.9
2 0 424832 4444 0 0 0 0 0 0 31 51286 2656 49 50 1 0 0.99 987.2
1 0 424930 4288 0 46 0 0 0 0 82 50550 2786 48 52 1 0 0.98 984.9
2 0 424971 4533 0 0 8 17 20 0 32 51066 2685 49 50 1 0 0.99 988.7
3 0 425029 4458 0 1 0 0 0 0 36 50424 2737 49 50 1 0 0.99 988.5
2 0 425096 4389 0 0 0 0 0 0 25 49992 2640 49 50 1 0 0.99 987.9
#AIX-Forum