Originally posted by: CristianLagos

I have a machine that's having a constant CPU load of 1, yet it isn't running any user processes (there's just patrol and a websphere nodeagent, but we have this machine as a backup of an active machine that is running websphere and that has a load of 0.69):

root@vm5-1:/home/root> oslevel -s
5300-05-04-0000
root@vm5-1:/home/root> uptime
04:35PM up 370 days, 21:13, 1 user, load average: 1.04, 1.05, 1.03
root@vm5-1:/home/root> vmstat -I 2 10

System configuration: lcpu=4 mem=6144MB ent=1.30

kthr memory page faults cpu

---

-----------

---

------------

---

r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
1 0 0 531869 630343 0 0 0 0 0 0 4 112 168 0 0 99 0 0.01 0.4
1 0 0 532092 630120 0 0 0 0 0 0 5 665 184 2 0 98 0 0.03 2.0
1 0 0 532430 629782 0 0 0 0 0 0 30 2856 307 1 1 97 0 0.04 2.9
1 0 0 532430 629782 0 3 0 0 0 0 10 749 184 1 0 98 0 0.02 1.6
1 0 0 532430 629782 0 0 0 0 0 0 4 40 167 0 0 99 0 0.00 0.3
1 0 0 532430 629782 0 0 0 0 0 0 3 33 168 0 0 99 0 0.00 0.3
1 0 0 532432 629780 0 0 0 0 0 0 4 197 180 0 0 99 0 0.01 0.5
1 0 0 531873 630338 0 0 0 0 0 0 15 3963 239 3 2 96 0 0.06 4.6
1 0 0 531874 630337 0 0 0 0 0 0 2 46 178 0 0 99 0 0.01 0.4
1 0 0 531873 630338 0 0 0 0 0 0 15 140 191 0 0 99 0 0.01 0.6

As seen, r (runnable processes), remain in 1 but it is not because of disk or network. We have another machine that is idle but it's not showing that behaviour:

root@vmpmq-1:/home/root> oslevel -s
5300-05-04-0000
root@vmpmq-1:/home/root> uptime
04:38PM up 273 days, 21:53, 1 user, load average: 0.01, 0.06, 0.12
root@vmpmq-1:/home/root> vmstat -I 2 10

System configuration: lcpu=2 mem=3072MB ent=0.60

kthr memory page faults cpu

---

-----------

---

------------

---

r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
0 0 0 303394 403496 0 1 0 0 0 0 7 210 172 0 1 99 0 0.01 1.1
0 0 0 303394 403496 0 0 0 0 0 0 4 85 159 1 0 99 0 0.01 1.4
0 0 0 303394 403496 0 0 0 0 0 0 6 956 176 2 2 96 0 0.03 4.6
0 0 0 303394 403496 0 0 0 0 0 0 2 222 168 2 1 97 0 0.02 3.3
0 0 0 303395 403494 0 0 0 0 0 0 4 290 163 0 1 99 0 0.01 1.0
0 0 0 303397 403492 0 0 0 0 0 0 4 1965 171 4 4 93 0 0.04 7.4
0 0 0 303397 403492 0 0 0 0 0 0 3 231 181 1 0 98 0 0.01 2.0
0 0 0 303400 403489 0 0 0 0 0 0 6 520 171 0 1 99 0 0.01 1.4
0 0 0 303402 403487 0 0 0 0 0 0 4 860 163 2 2 96 0 0.03 4.6
0 0 0 303404 403485 0 0 0 0 0 0 8 2462 183 1 5 93 0 0.04 7.1

How can I find the process that is keeping the load of the machine in 1?
Thanks in advance

Cristian
#AIX-Forum

Originally posted by: cd3lgado

Hi

You can try installing nmon program and make it capture snapshots of performance you can later review in order to look for the programs consuming your system's CPU resource.

http://www.ibm.com/developerworks/aix/library/au-analyze_aix/

HTH
#AIX-Forum

Originally posted by: Holgervk

tprof -k -x sleep 10
cat sleep.prof
#AIX-Forum

Originally posted by: CristianLagos

Thanks for the answers. I'll try with nmon
I don't see any clue using tprof:

Configuration information
=========================
System: AIX 5.3 Node: vm5-1 Machine: 00C3E3F04C00
Tprof command was:
tprof -k -x sleep 10
Trace command was:
/usr/bin/trace -ad -M -L 131866214 -T 500000 -j 000,00A,001,002,003,38F,005,006,134,139,5A2,5A5,465,234, -o -
Total Samples = 1301
Traced Time = 10.01s (out of a total execution time of 10.01s)
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Process Freq Total Kernel User Shared Other
======= ==== ===== ====== ==== ====== =====
wait 4 97.16 97.16 0.00 0.00 0.00
PatrolAgent 1 1.15 0.31 0.23 0.61 0.00
/usr/sbin/syncd 2 0.54 0.54 0.00 0.00 0.00
/usr/sbin/sshd 2 0.54 0.15 0.00 0.38 0.00
/usr/bin/ksh 2 0.15 0.15 0.00 0.00 0.00
/usr/bin/sh 2 0.15 0.15 0.00 0.00 0.00
/usr/bin/trcstop 1 0.08 0.00 0.00 0.08 0.00
bgsagent 1 0.08 0.08 0.00 0.00 0.00
/usr/bin/awk 1 0.08 0.08 0.00 0.00 0.00
swapper 1 0.08 0.08 0.00 0.00 0.00
======= ==== ===== ====== ==== ====== =====
Total 17 100.00 98.69 0.23 1.08 0.00

Process PID TID Total Kernel User Shared Other
======= === === ===== ====== ==== ====== =====
wait 8196 8197 74.10 74.10 0.00 0.00 0.00
wait 53274 81961 7.69 7.69 0.00 0.00 0.00
wait 61470 94255 7.69 7.69 0.00 0.00 0.00
wait 57372 86059 7.69 7.69 0.00 0.00 0.00
PatrolAgent 282636 15167581 1.15 0.31 0.23 0.61 0.00
/usr/sbin/sshd 13340904 15835185 0.46 0.08 0.00 0.38 0.00
/usr/sbin/syncd 90234 286869 0.38 0.38 0.00 0.00 0.00
/usr/sbin/syncd 90234 282771 0.15 0.15 0.00 0.00 0.00
/usr/bin/trcstop 13369572 11276325 0.08 0.00 0.00 0.08 0.00
/usr/bin/sh 13365270 15134897 0.08 0.08 0.00 0.00 0.00
bgsagent 295126 18804907 0.08 0.08 0.00 0.00 0.00
/usr/sbin/sshd 221314 524289 0.08 0.08 0.00 0.00 0.00
swapper 0 3 0.08 0.08 0.00 0.00 0.00
/usr/bin/awk 2666530 14975177 0.08 0.08 0.00 0.00 0.00
/usr/bin/sh 13324450 1061047 0.08 0.08 0.00 0.00 0.00
/usr/bin/ksh 13340906 15835187 0.08 0.08 0.00 0.00 0.00
/usr/bin/ksh 13324452 1061049 0.08 0.08 0.00 0.00 0.00
======= === === ===== ====== ==== ====== =====
Total 100.00 98.69 0.23 1.08 0.00

Total % For All Processes (KERNEL) = 98.62

Subroutine % Source
========== ====== ======
h_cede_end_point 96.93 hcalls.s
pcs_glue 0.31 vmvcs.s
.unlock_enable_mem 0.31 64/low.s
.iSyncNeeded 0.23 kernel/j2/j2_inode.c
.syncHashList 0.15 kernel/j2/j2_inode.c
.v_delscb 0.08 rnel/vmm/v_scblist.c
.v_mvfork 0.08 ernel/vmm/v_mvfork.c
.v_scan_vmp_size 0.08 /kernel/vmm/vmscan.c
.v_write_pseg@AF87_7 0.08 rnel/vmm/v_putsubs.c
.v_relalias 0.08 rnel/vmm/v_relsubs.c
.vreclaim 0.08 rnel/vmm/v_getsubs.c
.waitproc 0.08 rnel/proc/dispatch.c
.lock_done_mem 0.08 64/low.s
.umem_move 0.08 64/low.s

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
#AIX-Forum

Originally posted by: Holgervk

shutdown the patrol agent and look if that always running process in vmstat disappears
#AIX-Forum

Originally posted by: CristianLagos

I stopped the patrol agent and the CPU load remained in 1 so I restarted it
Using nmon I found that the swapper process is appearing as active (using t m 3 .):

âânmon12eâââââe=disk-ESSâââââââââHost=vm5-1ââââââââââRefresh=2 secsâââ11:21.04âââââââââââââââââââââââââââ
â Memory ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
â Physical PageSpace | pages/sec In Out | FileSystemCache â
â% Used 59.9% 4.7% | to Paging Space 0.0 0.0 | (numperm) 29.6% â
â% Free 40.1% 95.3% | to File System 0.0 0.0 | Process 9.3% â
âMB Used 3677.6MB 340.2MB | Page Scans 0.0 | System 20.9% â
âMB Free 2466.4MB 6827.8MB | Page Cycles 0.0 | Free 40.1% â
âTotal(MB) 6144.0MB 7168.0MB | Page Steals 0.0 |

---

â
â | Page Faults 382.8 | Total 100.0% â
â------------------------------------------------------------ | numclient 24.9% â
âMin/Maxperm 1176MB( 19%) 4705MB( 77%) <--% of RAM | maxclient 76.6% â
âMin/Maxfree 960 1088 Total Virtual 13.0GB | User 35.1% â
âMin/Maxpgahead 2 8 Accessed Virtual 2.0GB 15.5%| Pinned 19.7% â
â Top-Processes-(75) ââââââMode=3 1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmdsââââââââââââââââââââââââââââ
â PID %CPU Size Res Res Res Char RAM Paging Command â
â Used KB Set Text Data I/O Use io other repage â
â 282636 0.7 41648 41744 3724 38020 6 1% 0 246 0 PatrolAgent â
â 0 0.1 384 384 0 384 0 0% 0 0 0 Swapper â
âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ

I tried nmon in other machines and swapper doesn't appear.
I believe Swapper cannot be restarted, so the solution would be rebooting the machine?
#AIX-Forum

Originally posted by: MarkTaylor

the "r" col is number of threads that became runnable within the sample time .. are you sure its always the same thread ..

Take some samples .. and see what you get each time ..

echo "th *" | kdb | grep "RUN"

Do you know what the swapper process does ?

ref: http://www.ibm.com/developerworks/aix/library/au-aix5_cpu/

"Decaying the CPU usage

It is possible that a thread can get a priority so low that it never has a chance to run. This would occur if you use only Formulas 1 and 2 without a mechanism to push a thread's priority level back up.

When a thread runs with SCHED_OTHER, its priority is degraded for its use of CPU time. When it is not running and is waiting for its turn, AIX tries to regain its priority by "decaying" its CPU charges, about once a second. The rule is simple: A CPU-bound job should be assigned a lower priority to allow other jobs to run, but it should not be discriminated against to the point that it cannot finish itself. All threads' CPU charge is decayed based on a predefined factor of once per second, as follows:

New Charge C = (Old Charge C) * d / 32 (5)

A kernel process Swapper does this job. Once every second, Swapper wakes up and handles the CPU charge decaying for all the threads. The default decay factor is 0.5 or d=16, which "discounts" or "waives" half of the CPU charge.

With this mechanism, a CPU-intensive job accumulates CPU charge, gets to a lower priority level, and then advances to a much higher level at the end of a second. On the other hand, an I/O-intensive job does not vary its priority up and down as much, since it generally accumulates less CPU time."

This would lead me to believe that the swapper is doing its task and always decaying .. I can see Patrol agent above the swapper process in your nmon output below .. have you stopped this yet to see if that cleans this up ?

Also, think about what patrol agent does .. when everything on a system is fine, then it has little to do, but when its reporting on lots of events then its going to use more CPU .. there may be an underlying issue on this system, do the normal things like check for full filesystems and hardware error in errpt etc ...

HTH
Mark Taylor
#AIX-Forum

Originally posted by: CristianLagos

Thanks for your answer
I'm not sure that is always the same thread... that's one of the things I'm trying to find out (I've been working just 4 months with AIX. I have so much to learn)
Nothing weird with the output of the command:

root@vm5-1:/var/log> for i in 1 2 3 4 5 6 ; do date ; echo "th *" | kdb | grep "RUN" ; sleep 10 ; done
Wed Aug 24 12:45:05 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+028C00 652*kdb_64 RUN 28C0B1 052 1 0
Wed Aug 24 12:45:22 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+010000 256>grep RUN 10006B 03C 0 0
pvthread+028C00 652*kdb_64 RUN 28C0BD 053 0 0
Wed Aug 24 12:45:40 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+0ECC00 3788*kdb_64 RUN ECC049 05A 0 0
Wed Aug 24 12:45:57 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+028C00 652*kdb_64 RUN 28C0C7 050 0 0
pvthread+0E6F00 3695>grep RUN E6F0C5 03C 0 0
Wed Aug 24 12:46:14 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+028C00 652*kdb_64 RUN 28C0C9 048 0 0
Wed Aug 24 12:46:33 CLST 2011
pvthread+000200 2>wait RUN 002005 0FF 0 00000 0
pvthread+001400 20!wait RUN 014029 0FF 1 00001 0
pvthread+001500 21!wait RUN 01502B 0FF 2 00002 0
pvthread+001700 23!wait RUN 01702F 0FF 3 00003 0
pvthread+028C00 652*kdb_64 RUN 28C0CD 052 0 0

I tried it a couple of times, always with the same result
I did shut down the patrol agent and wait to see if the load dropped, but it didn't, so I restarted it
The error log is empty (errpt doesn't give any output)
I don't have any filesystem with problems
The CPU used is ok:

root@vm5-1:/var/log> sar 2 30 | grep -e idle -e Average
13:09:54 %usr %sys %wio %idle physc %entc
Average 1 1 0 98 0.02 1.3

Memory is also ok:

root@vm5-1:/var/log> svmon -G
size inuse free pin virtual
memory 1572864 940358 632506 310655 527340
pg space 1835008 86982

work pers clnt other
pin 251909 0 0 58746
in use 466868 77139 396351

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 889238 86982 282527 476220
m 64 KB - 3195 0 1758 3195
root@vm5-1:/var/log> vmstat -v
1572864 memory pages
1505574 lruable pages
632514 free pages
3 memory pools
310654 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
30.9 numperm percentage
466375 file pages
0.0 compressed percentage
0 compressed pages
26.0 numclient percentage
80.0 maxclient percentage
392319 client pages
0 remote pageouts scheduled
10267 pending disk I/Os blocked with no pbuf
10521 paging space I/Os blocked with no psbuf
143459 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
2105 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults

Also, there is no page in or page out:

root@vm5-1:/var/log> vmstat 2 10

System configuration: lcpu=4 mem=6144MB ent=1.30

kthr memory page faults cpu

---

-----------

---

------------

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 527351 632492 0 0 0 0 0 0 15 3780 233 3 1 96 0 0.06 4.2
1 0 527351 632491 0 0 0 0 0 0 24 304 250 1 0 99 0 0.02 1.2
1 0 527351 632491 0 0 0 0 0 0 5 182 161 1 0 99 0 0.01 0.8
1 0 527351 632491 0 0 0 0 0 0 3 33 162 0 0 99 0 0.00 0.4
1 0 527351 632491 0 0 0 0 0 0 29 112 196 0 2 97 0 0.03 2.3
1 0 527350 632491 0 0 0 0 0 0 2 87 162 0 0 99 0 0.00 0.3
1 0 527355 632486 0 0 0 0 0 0 13 2091 186 4 1 95 0 0.06 4.6
1 0 527356 632485 0 0 0 0 0 0 1 82 177 0 0 99 0 0.01 0.4
1 0 527356 632485 0 0 0 0 0 0 5 401 160 0 0 99 0 0.01 0.5
1 0 527357 632484 0 0 0 0 0 0 7 100 168 0 0 98 1 0.01 0.6

Yet the load keeps around 1 (and so the 'r' column):

root@vm5-1:/var/log> uptime
01:11PM up 375 days, 16:49, 1 user, load average: 1.00, 1.05, 1.11

So I'm still trying to catch that (or those) running thread
#AIX-Forum