Originally posted by: SystemAdmin

Hi Everyone
I have p550 with 4 Logical CPUs, 10GB Realmem, 3 146GB disks and one LPAR is created with min. 1 and max 4 CPUs.

When I do "sar -P ALL 2 10"
It shows that User% of one CPU is always 100% and other CPUs are mostly idle.
There is not much I/O activity on any disk, not much memory faults or paging.
20GB of Paging Space is given.

Statistics are as follows:

bash-2.04# sar -P ALL 3 30

AIX policy 3 5 00C8F8FF4C00 07/18/07

System configuration: lcpu=4

14:23:26 cpu %usr %sys %wio %idle physc
14:23:29 0 0 1 0 99 0.52
1 0 0 1 99 0.48
2 100 0 0 0 1.00
3 16 39 0 45 0.00
- 50 0 0 49 2.00
14:23:32 0 0 1 0 99 0.53
1 0 0 0 99 0.49
2 100 0 0 0 1.00
3 0 6 0 94 0.00
- 50 0 0 50 2.02
14:23:35 0 0 1 0 99 0.50
1 0 0 0 100 0.49
2 100 0 0 0 1.00
3 16 39 0 45 0.00
- 50 0 0 50 1.99
14:23:38 0 1 1 0 98 0.52
1 0 0 0 100 0.48
2 100 0 0 0 1.00
3 1 1 0 98 0.00
- 50 0 0 50 2.00
14:23:41 0 18 1 0 81 0.57
1 0 0 0 100 0.44
2 95 0 0 5 0.95
3 0 0 0 100 0.05
- 50 0 0 50 2.00
14:23:44 0 11 1 0 88 0.55
1 0 0 0 100 0.44
2 97 0 0 3 0.97
3 0 0 0 100 0.03
- 50 0 0 50 1.99
14:23:47 0 0 1 0 99 0.52
1 0 0 0 100 0.49
2 100 0 0 0 1.00
3 4 43 0 53 0.00
- 50 0 0 50 2.01
14:23:50 0 0 1 0 99 0.52
1 0 0 0 100 0.48
2 100 0 0 0 1.00
3 16 39 0 45 0.00
- 50 0 0 50 2.00
14:23:53 0 0 1 0 99 0.52
1 0 0 0 100 0.48
2 100 0 0 0 1.00
3 14 39 0 47 0.00
- 50 0 0 50 2.00
14:23:56 0 0 1 0 99 0.52
1 0 0 0 100 0.48
2 100 0 0 0 1.00
3 3 45 0 52 0.00
- 50 0 0 50 2.00
bash-2.04# vmstat -s
3476611252 total address trans. faults
985431208 page ins
275090145 page outs
5669802 paging space page ins
8679476 paging space page outs
0 total reclaims
135455565 zero filled pages faults
37944285 executable filled pages faults
1665311173 pages examined by clock
685 revolutions of the clock hand
583729105 pages freed by the clock
544391431 backtracks
2776577 free frame waits
0 extend XPT waits
49893406 pending I/O waits
1260513386 start I/Os
147499408 iodones
25242823104 cpu context switches
877848206 device interrupts
1479344486 software interrupts
13457245381 decrementer interrupts
7138345 mpc-sent interrupts
7138026 mpc-receive interrupts
7048976 phantom interrupts
0 traps
3257368691394 syscalls

bash-2.04# vmstat 2 5

System configuration: lcpu=4 mem=9792MB

kthr memory page faults cpu

---

-----------

---

------------

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 1900505 151143 0 0 0 0 0 0 10 293 269 50 0 49 0
1 0 1900506 151142 0 0 0 0 0 0 10 257 263 50 0 49 0
1 0 1900506 151142 0 0 0 0 0 0 15 142 254 50 0 50 0
1 0 1900506 151142 0 0 0 0 0 0 16 205 291 50 0 49 0
2 0 1900506 151142 0 0 0 0 0 0 9 195 273 50 0 49 0

bash-2.04# lparstat -i
Node Name : policy
Partition Name : policy
Partition Number : 1
Type : Dedicated-SMT
Mode : Capped
Entitled Capacity : 2.00
Partition Group-ID : 32769
Shared Pool ID : -
Online Virtual CPUs : 2
Maximum Virtual CPUs : 2
Minimum Virtual CPUs : 1
Online Memory : 9792 MB
Maximum Memory : 10240 MB
Minimum Memory : 128 MB
Variable Capacity Weight : -
Minimum Capacity : 1.00
Maximum Capacity : 2.00
Capacity Increment : 1.00
Maximum Physical CPUs in system : 2
Active Physical CPUs in system : 2
Active CPUs in Pool : -
Unallocated Capacity : -
Physical CPU Percentage : 100.00%
Unallocated Weight : -

bash-2.04# mpstat

System configuration: lcpu=4

cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc
0 644 0 0 227 251 149 0 305384256 100 17699 6 5 1 89 0.52
1 231 0 0 50 77 38 0 57229412 100 31045 8 6 0 87 0.48
2 647 0 0 225 520 272 0 294270706 100 45968 13 8 1 78 0.56
3 223 0 0 40 53 27 0 49388827 100 18140 6 4 0 90 0.44
ALL 1745 0 0 542 901 486 0 706273201 100 112852 8 6 0 86 2.00
Name PID CPU% PgSp Owner NFS (calls/sec) % Free 79.0
db2sysc 3436620 50.0 10.1 db2inst1 ServerV2 0
sadc 2097230 0.5 0.2 irl ClientV2 0 Press:
sadc 2990114 0.4 0.2 irl ServerV3 0 "h" for help
topas 3555468 0.3 2.9 root ClientV3 0 "q" to quit
java 2572540 0.1 457.6 ppolicy
java 2666624 0.0 181.1 ppolicy
java 2920612 0.0 21.7 irl
getty 245980 0.0 0.4 root
db2fmp 2891824 0.0 6.6 db2inst1
db2sysc 1663126 0.0 0.4 db2inst1
muxatmd 336102 0.0 0.6 root
rmcd 295068 0.0 223.6 root
db2fmd 2986168 0.0 1.1 dasusr1
sar 2773118 0.0 0.2 irl
sshd 3280984 0.0 0.9 smon

This is db2 process which is using most of the CPU time but only on one CPU.
Please suggest some tuning changes.

Regards
Ammi
#AIX-Forum

Originally posted by: MarkTaylor

Contrary to popular belief, db2 v8 is not multithreaded (1:1) so each application will only go one CPU (cpu affinity forgiving that is) and I am assuming v8 btw.

If you are spinning in user space, then its a application issue, so what you need to do is a "db2 list applications show detail" and grep for the suspect PID to get the db2 app that is causing the issue, then do a snapshot for it and db2 explain the SQL to see if it can be optimised.

At the end of the day this is a db2 issue, so you may want to post to a db2 group or get a dba to look at it :)

HTH
Mark Taylor
#AIX-Forum

Originally posted by: SystemAdmin

Hi Mark
Thanks for your reply, yes you are correct, I tried bindprocessor also but that also didnt work.
Actually, my db2 team is running a procedure to replace some records, which was working fine before but for the last few days,it has started giving problem. Actually the main problem is db2sysc process is always at 100% CPU utilization. What are the possible ways to fine tune a process (not processor) and monitor its activities.
I will update db2 team for that.
Actually the confusion comes when the team says the same process was working fine earlier.
Thanks Mark
Regards
Amit Bist
#AIX-Forum