Originally posted by: madunix
my AIX servers having high CPU utlization and performance problem,
I am looking to solve these performance bottleneck.
I am running 3 nodes with Oracle 10g rel2 based on
AIX version 5.3 maintenance level 3
prod2 /> oslevel -r
5300-03
Has 8Gb of memory with 8Gb set for swap. 2 CPU each node.
prod2 /> lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 8192MB 15 yes yes lv
.
root@prod2 /> vmstat 5 5
System configuration: lcpu=4 mem=7808MB
kthr memory page faults cpu
-----------
------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 1864549 37664 0 0 0 0 0 0 418 7691 1177 13 3 84 0
1 0 1878255 23954 0 0 0 0 0 0 407 8047 1156 16 4 79 1
1 0 1878179 24005 0 5 0 0 0 0 511 9008 1329 57 5 37 1
1 0 1865047 36929 0 9 0 0 0 0 518 8420 1378 13 4 81 1
2 0 1874914 27009 0 0 0 0 0 0 476 8339 1274 45 4 51 0
root@prod2 /> vmstat 5 100
System configuration: lcpu=4 mem=7808MB
kthr memory page faults cpu
-----------
------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 1879781 21825 0 0 0 0 0 0 305 13066 976 59 4 38 0
2 0 1882175 19429 0 0 0 0 0 0 422 13109 1188 62 3 34 0
2 0 1881781 19823 0 0 0 0 0 0 442 13801 1194 73 3 24 0
1 0 1881947 19655 0 0 0 0 0 0 338 13458 1009 60 3 37 0
1 0 1886156 15445 0 0 0 0 0 0 348 12081 1007 62 3 35 0
3 0 1887608 20775 0 9 1330 1365 2306 0 1323 109627 3104 71 8 21 0
2 0 1892745 15636 0 0 0 0 0 0 320 11230 937 61 3 36 0
1 0 1881703 26678 0 0 0 0 0 0 1610 16649 3552 60 5 35 0
1 0 1885272 23108 0 0 0 0 0 0 1416 27812 3174 62 6 32 0
1 0 1883491 24883 0 0 0 0 0 0 1284 9381 2910 52 5 43 0
1 0 1870183 38191 0 0 0 0 0 0 392 10206 1149 67 5 28 0
1 0 1872259 36111 0 0 0 0 0 0 505 9143 1362 40 4 55 2
2 1 1883201 25168 0 0 0 0 0 0 270 10897 918 49 4 47 0
2 0 1878940 29405 0 4 0 0 0 0 446 11649 1183 67 4 29 0
1 0 1873749 34592 0 0 0 0 0 0 309 11547 922 67 4 29 0
2 0 1884657 23682 0 0 0 0 0 0 371 11625 1024 59 4 37 0
1 0 1873743 34596 0 0 0 0 0 0 580 12621 1668 58 4 38 0
1 0 1878815 29518 0 1 0 0 0 0 377 12032 1146 58 3 39 0
2 0 1884958 23373 0 0 0 0 0 0 420 53484 1282 69 5 26 0
2 0 1877442 30888 0 0 0 0 0 0 317 9119 956 42 3 54 0
kthr memory page faults cpu
-----------
------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 1873240 35054 0 4 0 0 0 0 315 21648 985 19 4 77 1
1 0 1873240 35054 0 0 0 0 0 0 321 6245 985 8 3 90 0
2 0 1873500 34792 0 0 0 0 0 0 304 7644 990 8 3 88 1
1 0 1876009 32282 0 0 0 0 0 0 220 7633 829 27 3 69 0
^C
root@prod2 /> iostat 5 5
System configuration: lcpu=4 drives=14 paths=0 vdisks=0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 692.8 53.4 3.1 43.4 0.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
dac0 0.0 2.8 3.0 12 2
dac0-utm 0.0 0.0 0.0 0 0
hdisk0 0.2 3.2 0.8 8 8
dac2 0.0 30.2 4.2 72 80
dac2-utm 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 11.1 2.8 8 48
hdisk5 0.2 2.8 3.0 12 2
hdisk6 0.0 19.1 1.4 64 32
hdisk7 0.0 0.0 0.0 0 0
hdisk8 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 1124.9 73.8 3.3 22.8 0.0
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
dac0 0.0 2.8 3.0 12 2
dac0-utm 0.0 0.0 0.0 0 0
hdisk0 0.0 2.4 0.6 4 8
dac2 0.0 62.3 6.2 104 209
dac2-utm 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 0.0 2.8 3.0 12 2
hdisk6 0.2 62.3 6.2 104 209
hdisk7 0.0 0.0 0.0 0 0
hdisk8 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 1064.0 69.7 2.5 27.7 0.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
dac0 0.0 2.8 3.0 12 2
dac0-utm 0.0 0.0 0.0 0 0
hdisk0 0.4 4.0 1.0 12 8
dac2 0.0 71.2 6.4 136 220
dac2-utm 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 0.0 2.8 3.0 12 2
hdisk6 0.8 71.2 6.4 136 220
hdisk7 0.0 0.0 0.0 0 0
hdisk8 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 1183.4 73.5 3.0 23.4 0.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
dac0 0.0 2.8 3.0 12 2
dac0-utm 0.0 0.0 0.0 0 0
hdisk0 1.0 8.0 2.0 28 12
dac2 0.0 36.8 3.4 128 56
dac2-utm 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 0.0 2.8 3.0 12 2
hdisk6 0.2 36.8 3.4 128 56
hdisk7 0.0 0.0 0.0 0 0
hdisk8 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
System configuration: type=Dedicated mode=Capped smt=On lcpu=4 mem=7808
%user %sys %wait %idle
----
7.5 3.0 0.4 89.1
root@prod2 /> vmstat -v
1998848 memory pages
1921916 lruable pages
41106 free pages
1 memory pools
169322 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
14.9 numperm percentage
287543 file pages
0.0 compressed percentage
0 compressed pages
15.8 numclient percentage
80.0 maxclient percentage
303763 client pages
0 remote pageouts scheduled
1743 pending disk I/Os blocked with no pbuf
162963 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
5086 client filesystem I/Os blocked with no fsbuf
4158 external pager filesystem I/Os blocked with no fsbuf
prod2] /> smtctl
This system is SMT capable.
SMT is currently enabled.
SMT boot mode is not set.
SMT threads are bound to the same physical processor.
proc0 has 2 SMT threads.
Bind processor 0 is bound with proc0
Bind processor 1 is bound with proc0
proc2 has 2 SMT threads.
Bind processor 2 is bound with proc2
Bind processor 3 is bound with proc2
#AIX-Forum