AIX

 View Only
Expand all | Collapse all

Problem - High load average, freeze service MUMPS/MSM

  • 1.  Problem - High load average, freeze service MUMPS/MSM

    Posted Tue June 14, 2022 03:06 PM
    Hello, today we had a new freeze in the system occurred at 15:48. Follow logs for analysis, does anyone suggest something?

    [Jun-14-22 15:48:10] Begin MSM process: ./mutio, id=1137, Proc#=27000906
    [Jun-14-22 15:48:16] Begin MSM process: ./mutio, id=1164, Proc#=43909158
    [Jun-14-22 15:48:16] Begin MSM process: ./mutio, id=1152, Proc#=37748990
    [Jun-14-22 15:48:17] Begin MSM process: ./mutio, id=1155, Proc#=30212246
    [Jun-14-22 15:48:18] E mumsm ended (proc=#18874578, status=90009)
    .................... E ...abnormal termination, signal # = 9
    [Jun-14-22 15:48:18] mutio ended (proc=#30212246, status=0)
    [Jun-14-22 15:48:18] mutio ended (proc=#37748990, status=0)


    kthr memory page faults cpu time
    ----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
    r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
    1 0 0 1385719 43193 0 0 0 0 0 0 141 6637 354 51 2 47 0 1.06 53.1 15:48:00
    1 0 0 1386034 42878 0 0 0 0 0 0 128 29468 316 51 2 47 0 1.06 53.2 15:48:02
    2 0 0 1386212 42697 0 0 0 0 0 0 123 86539 361 56 5 39 0 1.22 61.0 15:48:04
    1 0 0 1386775 42134 0 0 0 0 0 0 121 17517 351 51 2 47 0 1.06 53.1 15:48:06
    1 0 0 1386858 42050 0 0 0 0 0 0 143 16048 444 52 2 46 0 1.08 54.0 15:48:08
    1 0 0 1386921 41985 0 0 0 0 0 0 139 27751 384 51 2 46 0 1.07 53.7 15:48:10
    1 0 0 1387056 41850 0 0 0 0 0 0 126 65413 345 54 2 43 0 1.13 56.7 15:48:12
    2 0 0 1387155 41751 0 0 0 0 0 0 142 16659 391 52 4 44 0 1.13 56.5 15:48:14
    1 0 0 1387584 41318 2 80 0 0 0 0 212 19417 516 50 5 45 0 1.10 55.2 15:48:16
    479 0 0 1037085 391802 0 2 0 0 0 0 629 42428 1217 36 22 42 0 1.17 58.6 15:48:18
    0 0 0 1026026 402850 0 0 0 0 0 0 174 5511 404 2 2 96 0 0.10 4.8 15:48:20
    5 0 0 991885 436944 1 15 0 0 0 0 228 29688 458 6 5 90 0 0.21 10.7 15:48:22
    4 0 0 836727 592097 2 0 0 0 0 0 191 29146 438 5 11 84 0 0.33 16.5 15:48:24
    0 0 0 836150 592679 0 0 0 0 0 0 195 18439 395 4 3 93 0 0.14 7.2 15:48:26
    0 0 0 836636 592191 0 0 0 0 0 0 221 12267 329 3 2 94 0 0.12 6.0 15:48:28
    0 0 0 837268 591556 0 0 0 0 0 0 271 48926 446 7 4 89 0 0.23 11.7 15:48:30
    1 0 0 838215 590607 0 0 0 0 0 0 303 33009 473 5 3 92 0 0.18 8.8 15:48:32
    0 0 0 838086 590733 0 0 0 0 0 0 260 74342 370 5 3 92 0 0.17 8.7 15:48:34
    1 0 0 1004499 424653 2 0 0 0 0 0 210 59347 529 45 7 48 0 1.05 52.4 15:48:36
    2 0 0 1073692 355437 0 0 0 0 0 0 197 158030 1402 25 31 44 0 1.12 56.2 15:48:38



    SISMED1:/var/tmp> vmstat -v output
    4194304 memory pages
    3993728 lruable pages
    46705 free pages
    1 memory pools
    744730 pinned pages
    90.0 maxpin percentage
    3.0 minperm percentage
    90.0 maxperm percentage
    68.8 numperm percentage
    2748310 file pages
    0.0 compressed percentage
    0 compressed pages
    68.8 numclient percentage
    90.0 maxclient percentage
    2748310 client pages
    0 remote pageouts scheduled
    1438 pending disk I/Os blocked with no pbuf
    0 paging space I/Os blocked with no psbuf
    2484 filesystem I/Os blocked with no fsbuf
    0 client filesystem I/Os blocked with no fsbuf
    23809 external pager filesystem I/Os blocked with no fsbuf
    33.4 percentage of memory used for computational pages




    SISMED1:/var/tmp/tprof_ps> cat sleep.prof
    Configuration information
    =========================
    System: AIX 7.1 Node: SISMED1 Machine: 00FB42D74C00
    Tprof command was:
    tprof -skex sleep 60
    Trace command was:
    /usr/bin/trace -ad -M -L 801994752 -T 500000 -j 00A,001,002,003,38F,005,006,134,210,139,5A2,5A5,465,234,5D8, -o -
    Total Samples = 12002
    Traced Time = 60.03s (out of a total execution time of 60.03s)
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

    Process Freq Total Kernel User Shared Other
    ======= ==== ===== ====== ==== ====== =====
    mumsm 1 49.48 0.10 49.38 0.00 0.00
    wait 1 47.35 47.35 0.00 0.00 0.00
    /usr/bin/topas 2 0.77 0.45 0.13 0.18 0.00
    /usr/bin/ps 13 0.67 0.54 0.01 0.12 0.00
    /usr/es/sbin/cluster/clstat 5 0.38 0.27 0.00 0.12 0.00
    /usr/sbin/zabbix_agentd: 16 0.23 0.15 0.03 0.05 0.00
    /usr/sbin/sshd 3 0.19 0.10 0.06 0.03 0.00
    /usr/bin/sh 9 0.07 0.05 0.01 0.02 0.00
    /usr/bin/login 3 0.06 0.03 0.00 0.02 0.00
    /bin/ksh 7 0.06 0.06 0.00 0.00 0.00
    /opt/rsct/bin/rmcd 1 0.06 0.02 0.00 0.04 0.00
    /usr/bin/awk 5 0.05 0.05 0.00 0.00 0.00
    /usr/bin/netstat 4 0.05 0.04 0.00 0.01 0.00
    lock_rcv 1 0.05 0.05 0.00 0.00 0.00
    /usr/sbin/syncd 2 0.04 0.04 0.00 0.00 0.00
    swapper 1 0.04 0.04 0.00 0.00 0.00
    -ksh 4 0.04 0.03 0.01 0.00 0.00
    /usr/sbin/snmpd 1 0.02 0.02 0.00 0.00 0.00
    /usr/sbin/getty 1 0.02 0.02 0.00 0.00 0.00
    /usr/bin/w 1 0.02 0.02 0.00 0.01 0.00
    /usr/bin/grep 3 0.02 0.01 0.00 0.02 0.00
    /bin/perl 2 0.02 0.02 0.00 0.01 0.00
    /usr/es/sbin/cluster/clinfo 1 0.02 0.02 0.00 0.01 0.00
    /usr/sbin/telnetd 2 0.02 0.02 0.00 0.00 0.00
    /usr/bin/head 2 0.02 0.02 0.00 0.00 0.00
    /usr/bin/tprof 1 0.02 0.00 0.02 0.00 0.00
    /var/opt/tivoli/ep/_jvm/jre/bin/java 2 0.02 0.02 0.00 0.00 0.00
    /usr/bin/wc 2 0.02 0.01 0.01 0.00 0.00
    /usr/es/sbin/cluster/clstrmgr 1 0.02 0.01 0.00 0.01 0.00
    /usr/sbin/aso 1 0.01 0.01 0.00 0.00 0.00
    /usr/sbin/rsct/bin/hagsd 1 0.01 0.01 0.00 0.00 0.00
    /usr/bin/date 1 0.01 0.01 0.00 0.00 0.00
    /usr/bin/cat 1 0.01 0.01 0.00 0.00 0.00
    /opt/rsct/bin/IBM.ServiceRMd 1 0.01 0.00 0.00 0.01 0.00
    /usr/bin/hostname 1 0.01 0.01 0.00 0.00 0.00
    /opt/rsct/bin/IBM.MgmtDomainRMd 1 0.01 0.01 0.00 0.00 0.00
    /usr/sbin/cron 1 0.01 0.01 0.00 0.00 0.00
    muctrl 1 0.01 0.01 0.00 0.00 0.00
    ./msm 1 0.01 0.01 0.00 0.00 0.00
    gil 1 0.01 0.01 0.00 0.00 0.00
    mutio 1 0.01 0.00 0.00 0.01 0.00
    reaffin 1 0.01 0.01 0.00 0.00 0.00
    rpc.lockd 1 0.01 0.01 0.00 0.00 0.00
    sshd: 1 0.01 0.01 0.00 0.00 0.00
    /bin/awk 1 0.01 0.01 0.00 0.00 0.00
    ======= ==== ===== ====== ==== ====== =====
    Total 113 100.00 49.68 49.66 0.67 0.00

    Process PID TID Total Kernel User Shared Other
    ======= === === ===== ====== ==== ====== =====
    mumsm 18874578 60686561 49.48 0.10 49.38 0.00 0.00
    wait 131076 131077 47.35 47.35 0.00 0.00 0.00
    /usr/bin/topas 23396568 106561767 0.50 0.27 0.06 0.17 0.00
    /usr/bin/topas 22151366 86180069 0.27 0.18 0.07 0.01 0.00
    /usr/sbin/sshd 24641908 86442165 0.12 0.07 0.02 0.03 0.00
    /sbin/cluster/clstat 30998636 88277017 0.09 0.07 0.00 0.02 0.00
    /sbin/zabbix_agentd: 7340264 11862131 0.08 0.03 0.02 0.02 0.00
    /sbin/cluster/clstat 8323446 81723641 0.07 0.05 0.00 0.02 0.00
    /sbin/cluster/clstat 8323452 81723647 0.07 0.05 0.00 0.02 0.00
    /sbin/cluster/clstat 54001794 71499893 0.07 0.05 0.00 0.02 0.00
    /sbin/cluster/clstat 6422948 43712633 0.07 0.05 0.00 0.02 0.00
    /usr/bin/ps 24641904 86442161 0.06 0.05 0.00 0.01 0.00
    /opt/rsct/bin/rmcd 6619338 11206743 0.06 0.02 0.00 0.04 0.00
    /usr/bin/ps 54001796 71499895 0.06 0.04 0.00 0.02 0.00
    /usr/bin/ps 8781940 64094315 0.06 0.05 0.00 0.01 0.00
    /usr/bin/ps 30998650 88277031 0.06 0.05 0.00 0.01 0.00
    /usr/bin/ps 8781938 64094313 0.05 0.05 0.00 0.00 0.00
    /usr/bin/ps 30998596 88277233 0.05 0.04 0.00 0.01 0.00
    /usr/bin/ps 8781912 64094287 0.05 0.02 0.01 0.02 0.00
    /usr/bin/ps 8781914 64094289 0.05 0.03 0.00 0.02 0.00
    lock_rcv 3145842 3997839 0.05 0.05 0.00 0.00 0.00
    /usr/bin/ps 6422964 43712649 0.05 0.04 0.00 0.01 0.00
    /usr/bin/ps 24641906 86442163 0.05 0.04 0.00 0.01 0.00
    /usr/bin/ps 30998594 88277231 0.05 0.05 0.00 0.00 0.00
    /usr/bin/ps 30998648 88277029 0.05 0.04 0.00 0.01 0.00
    /usr/bin/login 26149154 11665557 0.04 0.02 0.00 0.02 0.00
    /usr/sbin/sshd 14549096 52756617 0.04 0.03 0.01 0.00 0.00
    swapper 0 3 0.04 0.04 0.00 0.00 0.00
    /usr/bin/ps 8781898 64094273 0.04 0.02 0.00 0.02 0.00
    /usr/sbin/sshd 14549094 52756615 0.03 0.00 0.03 0.00 0.00
    /usr/sbin/syncd 2228388 5177503 0.03 0.03 0.00 0.00 0.00
    /sbin/zabbix_agentd: 7209182 11731049 0.02 0.02 0.00 0.00 0.00
    /sbin/cluster/clinfo 12320928 32571423 0.02 0.02 0.00 0.01 0.00
    /usr/sbin/getty 4915392 9961623 0.02 0.02 0.00 0.00 0.00
    /usr/sbin/snmpd 5570740 9109527 0.02 0.02 0.00 0.00 0.00
    /usr/bin/w 59048148 84934777 0.02 0.02 0.00 0.01 0.00
    /usr/bin/tprof 20250844 29687955 0.02 0.00 0.02 0.00 0.00
    /usr/sbin/telnetd 24641892 86442147 0.02 0.02 0.00 0.00 0.00
    /usr/bin/awk 11862520 62914615 0.02 0.02 0.00 0.00 0.00
    bin/cluster/clstrmgr 7405600 12255355 0.02 0.01 0.00 0.01 0.00
    /sbin/zabbix_agentd: 59506858 51904669 0.02 0.02 0.00 0.00 0.00
    /bin/perl 8323434 81723629 0.02 0.01 0.00 0.01 0.00
    /usr/bin/netstat 20906316 44761197 0.02 0.02 0.00 0.00 0.00
    /usr/bin/netstat 8323432 81723627 0.02 0.01 0.00 0.01 0.00
    -ksh 30998660 88277041 0.02 0.02 0.00 0.00 0.00
    /bin/ksh 11862504 62914599 0.01 0.01 0.00 0.00 0.00
    /bin/awk 24641890 46334183 0.01 0.01 0.00 0.00 0.00
    ./msm 26149154 11665557 0.01 0.01 0.00 0.00 0.00
    -ksh 26149154 11665557 0.01 0.00 0.01 0.00 0.00
    -ksh 8781900 64094275 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 30998602 88277239 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 30998592 88277229 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 30998590 88277227 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 54001816 71499915 0.01 0.00 0.01 0.00 0.00
    /usr/bin/sh 30998614 88276995 0.01 0.01 0.00 0.00 0.00
    -ksh 11862512 62914607 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 30474562 100728951 0.01 0.01 0.00 0.00 0.00
    /usr/bin/awk 30998592 88277229 0.01 0.01 0.00 0.00 0.00
    /usr/bin/sh 24641890 46334183 0.01 0.00 0.00 0.01 0.00
    /usr/bin/sh 398 89129073 0.01 0.00 0.00 0.01 0.00
    /usr/bin/sh 30998638 88277019 0.01 0.01 0.00 0.00 0.00
    /usr/bin/awk 30998606 88277243 0.01 0.01 0.00 0.00 0.00
    /usr/bin/cat 11862512 62914607 0.01 0.01 0.00 0.00 0.00
    /usr/bin/date 30998618 88276999 0.01 0.01 0.00 0.00 0.00
    /usr/bin/grep 20906312 44761193 0.01 0.00 0.00 0.01 0.00
    /usr/bin/wc 8781926 64094301 0.01 0.01 0.00 0.00 0.00
    /usr/bin/wc 8781904 64094279 0.01 0.00 0.01 0.00 0.00
    /usr/bin/grep 11862516 62914611 0.01 0.00 0.00 0.01 0.00
    /usr/bin/grep 8323420 81723615 0.01 0.01 0.00 0.00 0.00
    /usr/bin/head 38994138 40697973 0.01 0.01 0.00 0.00 0.00
    /usr/bin/head 6422944 43712629 0.01 0.01 0.00 0.00 0.00
    /usr/bin/hostname 30998626 88277007 0.01 0.01 0.00 0.00 0.00
    /usr/bin/login 14549088 52756609 0.01 0.01 0.00 0.00 0.00
    /usr/bin/awk 24183094 68878449 0.01 0.01 0.00 0.00 0.00
    /usr/sbin/aso 4522136 7864575 0.01 0.01 0.00 0.00 0.00
    /usr/sbin/cron 11862504 62914599 0.01 0.01 0.00 0.00 0.00
    /usr/bin/login 10289490 95223893 0.01 0.00 0.00 0.01 0.00
    /sbin/rsct/bin/hagsd 8061182 24641541 0.01 0.01 0.00 0.00 0.00
    /usr/bin/netstat 30474558 100728947 0.01 0.01 0.00 0.00 0.00
    /usr/bin/netstat 59048152 84934781 0.01 0.01 0.00 0.00 0.00
    /usr/bin/awk 24183112 68878467 0.01 0.01 0.00 0.00 0.00
    t/bin/IBM.ServiceRMd 7602424 18284591 0.01 0.00 0.00 0.01 0.00
    /usr/sbin/syncd 2228388 5111965 0.01 0.01 0.00 0.00 0.00
    in/IBM.MgmtDomainRMd 4980862 17039369 0.01 0.01 0.00 0.00 0.00
    /usr/sbin/telnetd 19267756 116588611 0.01 0.01 0.00 0.00 0.00
    /bin/perl 30998610 88277247 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 24641896 86442151 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 24641902 86442159 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 30998632 88277013 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 8781936 64094311 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 30998598 88277235 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 54001818 71499917 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 8781932 64094307 0.01 0.00 0.00 0.01 0.00
    /sbin/zabbix_agentd: 8781924 64094299 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 30998654 88277035 0.01 0.01 0.00 0.00 0.00
    /bin/ksh 8781914 64094289 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 30998622 88277003 0.01 0.01 0.00 0.00 0.00
    /bin/ksh 24183100 68878455 0.01 0.01 0.00 0.00 0.00
    /sbin/zabbix_agentd: 6422942 43712627 0.01 0.00 0.00 0.01 0.00
    /sbin/zabbix_agentd: 8781920 64094295 0.01 0.00 0.01 0.00 0.00
    /sbin/zabbix_agentd: 30998588 88277225 0.01 0.00 0.00 0.01 0.00
    /bin/ksh 24183110 68878465 0.01 0.01 0.00 0.00 0.00
    ep/_jvm/jre/bin/java 7995446 25428021 0.01 0.01 0.00 0.00 0.00
    ep/_jvm/jre/bin/java 7995446 20381815 0.01 0.01 0.00 0.00 0.00
    gil 1179684 2031679 0.01 0.01 0.00 0.00 0.00
    /bin/ksh 8781912 64094287 0.01 0.01 0.00 0.00 0.00
    muctrl 52101226 117637349 0.01 0.01 0.00 0.00 0.00
    /bin/ksh 8781910 64094285 0.01 0.01 0.00 0.00 0.00
    mutio 8716736 37552289 0.01 0.00 0.00 0.01 0.00
    reaffin 589842 1441837 0.01 0.01 0.00 0.00 0.00
    rpc.lockd 4063376 8978455 0.01 0.01 0.00 0.00 0.00
    sshd: 20906294 44761175 0.01 0.01 0.00 0.00 0.00
    /bin/ksh 8781946 64094321 0.01 0.01 0.00 0.00 0.00
    ======= === === ===== ====== ==== ====== =====
    Total 100.00 49.68 49.66 0.67 0.00

    Total % For All Processes (KERNEL) = 49.56

    Subroutine % Source
    ========== ====== ======
    h_cede_end_point 47.14 hcalls.s
    .unlock_enable_mem 0.44 low.s
    ovlya_addr_pcs_glue_xon2 0.22 vmvcs.s
    sc_msr_2_point 0.13 low.s
    .getthrds 0.09 ernel/proc/getproc.c
    svc_rfid_ret32 0.08 low.s
    .v_lookup_mpss 0.07 ernel/vmm/v_lookup.c
    .simple_unlock_mem 0.07 low.s
    .enable 0.06 misc.s
    hypervisor_call_asm_end_point 0.05 hcalls.s
    svc_rfid_ret64 0.04 low.s
    .kernel_add_gate_cstack 0.04 low.s
    ._hkeyset_restore_userkeys 0.04 64/skeys.s
    .fill_procentry64 0.04 ernel/proc/getproc.c
    .syncHashList 0.03 kernel/j2/j2_inode.c
    .simple_lock 0.03 low.s
    .disable_lock 0.03 low.s
    .v_lookup 0.02 ernel/vmm/v_lookup.c
    hk_update_ukeys_accr_point 0.02 64/skeys.s
    .hkey_legacy_gate 0.02 low.s
    .fetch_and_addlp 0.02 low.s
    .sc_debug_fixup 0.02 low.s
    .hkeyset_add 0.02 low.s
    .v_vpgcount 0.02 rnel/vmm/v_segsubs.c
    .v_scan_vmp 0.02 /kernel/vmm/vmscan.c
    .pfget 0.02 rnel/vmm/v_getsubs.c
    .trchook64 0.02 trchka64.s
    vcs_p2_point 0.02 vmvcs.s
    .v_reclaim 0.02 rnel/vmm/v_getsubs.c
    .hkeyset_restore 0.02 low.s
    .v_delpft 0.02 kernel/vmm/v_lists.c
    .umem_move 0.02 low.s
    .kernel_replace_gate_cstack 0.01 low.s
    .drw_lock_read 0.01 low.s
    .as_getadsp64 0.01 /kernel/vmm/vmadsp.c
    .crxref 0.01 kernel/s/auth/cred.c
    begbt 0.01 vmvcs.s
    begbt_pretime_point 0.01 vmvcs.s
    begfst_pretime_point 0.01 vmvcs.s
    .waitproc_find_run_queue 0.01 rnel/proc/dispatch.c
    .get_next_pvproc 0.01 kernel/proc/pmzone.c
    .copyin 0.01 /kernel/vmm/userio.c
    _hkeyset_update_userkeys 0.01 64/skeys.s
    .mtrchook5 0.01 low.s
    .mtrchook2 0.01 low.s
    .lock_read 0.01 low.s
    .convert_tb 0.01 ernel/proc/curtime.c
    .put_onto_list 0.01 misc_ppc.s
    .xmattach 0.01 os/kernel/vmm/xmem.c
    .v_wpagein 0.01 nel/vmm/v_getsubs1.c
    .v_pf_dodisk 0.01 nel/vmm/v_getsubs1.c
    .v_ff_bitmap_head_tail 0.01 nel/vmm/v_freelist.c
    .v_getframe 0.01 nel/vmm/v_freelist.c
    .vm_scb_reaffin_kproc 0.01 ernel/vmm/vm_affin.c
    .simple_lock_try 0.01 low.s
    .compare_and_swap 0.01 low.s
    .v_sort_wsidlist 0.01 rnel/vmm/v_relsubs.c
    .v_releasew_getxpt 0.01 rnel/vmm/v_relsubs.c
    .pfget_mmap 0.01 rnel/vmm/v_getsubs.c
    ._ptrgl 0.01 low.s
    .v_findxpt 0.01 rnel/vmm/v_xptsubs.c
    .spec_getattr 0.01 /specfs/spec_vnops.c
    .allocxpt 0.01 rnel/vmm/v_xptsubs.c
    .v_delscb 0.01 rnel/vmm/v_scblist.c
    .v_free_reservation 0.01 rnel/vmm/v_spaceok.c
    .v_use_reservation 0.01 rnel/vmm/v_spaceok.c
    .v_cleardata 0.01 rnel/vmm/v_segsubs.c
    .v_scan_end 0.01 /kernel/vmm/vmscan.c
    .v_scan_new 0.01 /kernel/vmm/vmscan.c
    .v_getattach 0.01 /kernel/vmm/v_numa.c
    .kern_geth 0.01 nel/vmm/vmadsputil.c
    .dtReadSort 0.01 kernel/j2/j2_dtree.c
    .kcred_getpag64 0.01 /kernel/s/auth/pag.c
    .j2_rele@AF24_6 0.01 /kernel/j2/j2_rele.c
    .j2_seek 0.01 /kernel/j2/j2_seek.c
    .v_gethandle 0.01 el/vmm/v_getsubs64.c
    .rt_texit@AF42_26 0.01 ernel/proc/rttimer.c
    sc_exit_etrc_point 0.01 low.s
    .getthem 0.01 ernel/proc/getproc.c
    .sys_call_ret 0.01 low.s
    .netinfo_unixdomnlist 0.01 /bos/kernel/net/if.c
    .rdwr 0.01 os/kernel/lfs/rdwr.c
    .dac_refmon 0.01 /refmon/dac_refmon.c
    .vm_disclaim_com 0.01 rnel/vmm/vmrelease.c
    .vms_set_mpolicy 0.01 kernel/vmm/vm_numa.c
    .ufdrele 0.01 /bos/kernel/lfs/fd.c
    .vnop_fid 0.01 s/kernel/lfs/vnops.c
    ._vmgetinfo 0.01 rnel/vmm/vmgetinfo.c
    .nlcLookup 0.01 bos/kernel/lfs/nlc.c

    Millicode Subroutine % Source
    ========== ====== ======
    .memset_overlay 0.07 low.s
    .memmove_overlay 0.04 low.s
    .strcmp_overlay 0.03 low.s
    .memmove64_overlay 0.02 low.s
    .memset64_overlay 0.02 low.s
    .strcmp64_overlay 0.01 low.s
    .memccpy64_overlay 0.01 low.s
    .bzero64_overlay 0.01 low.s

    Total % For All Processes (KEX) = 0.12

    Kernel Ext %
    ========== ======
    /usr/lib/drivers/netinet 0.04
    /usr/lib/drivers/vioentdd 0.03
    /usr/lib/perf/perfstat 0.02
    UnknownBinary 0.02
    /usr/lib/drivers/ldterm 0.01

    Profile: /usr/lib/drivers/netinet

    Total % For All Processes (/usr/lib/drivers/netinet) = 0.04

    Subroutine % Source
    ========== ====== ======
    .ipintr_noqueue_post_fw 0.02 /lib/drivers/netinet
    .tcp_usrreq 0.01 /lib/drivers/netinet
    .pmtu_walktable 0.01 /lib/drivers/netinet
    .m_free 0.01 glink64.s

    Profile: /usr/lib/drivers/vioentdd

    Total % For All Processes (/usr/lib/drivers/vioentdd) = 0.03

    Subroutine % Source
    ========== ====== ======
    .vioent_receive 0.02 ent/vioent_receive.c
    .vioent_output 0.01 oent/vioent_output.c
    .getcaller 0.01 glink64.s

    Profile: /usr/lib/perf/perfstat

    Total % For All Processes (/usr/lib/perf/perfstat) = 0.02

    Subroutine % Source
    ========== ====== ======
    .s_update_myDkstats 0.01 sr/lib/perf/perfstat
    ._ptx_get_numifnets 0.01 sr/lib/perf/perfstat

    Profile: UnknownBinary

    Total % For All Processes (UnknownBinary) = 0.02

    Subroutine % Source
    ========== ====== ======
    <0xF1000000C03E7094> 0.02

    Profile: /usr/lib/drivers/ldterm

    Total % For All Processes (/usr/lib/drivers/ldterm) = 0.01

    Subroutine % Source
    ========== ====== ======
    .ldtty_output 0.01 r/lib/drivers/ldterm

    Total % For All Processes (SH-LIBs) = 0.67

    Shared Object %
    ============= ======
    /usr/lib/libc.a[shr.o] 0.23
    /usr/lib/libc.a[shr_64.o] 0.17
    /usr/lib/libperfstat.a[shr.o] 0.11
    /usr/lib/libpthreads.a[shr_xpg5.o] 0.05
    /usr/lib/libct_mss.a[shr.o] 0.02
    /usr/lib/libpthreads.a[shr.o] 0.02
    /usr/bin/ps 0.02
    /usr/lib/libcrypto.a[libcrypto.so.1.0.0] 0.01
    /usr/lib/libct_skc.a[shr.o] 0.01
    /usr/lib/libC.a[shr.o] 0.01
    /usr/opt/perl5/lib/5.28.1/aix-thread-multi/CORE/libperl.a[libperl.o] 0.01
    /usr/lib/libct_rm.a[shr.o] 0.01

    Profile: /usr/lib/libc.a[shr.o]

    Total % For All Processes (/usr/lib/libc.a[shr.o]) = 0.23

    Subroutine % Source
    ========== ====== ======
    .strpbrk 0.03 s/lib/libc/strpbrk.c
    .sv_next 0.02 /lib/libirs/lcl_sv.c
    ._doprnt 0.02 cs/lib/libc/doprnt.c
    .__regexec_std 0.01 libc/__regexec_std.c
    .sv_byname 0.01 /lib/libirs/lcl_sv.c
    .sendto 0.01 lib/libc/net_stubs.c
    .time 0.01 ib/libc/POWER/time.c
    .getsystemcfg 0.01 /libc/getsystemcfg.c
    .getuidx 0.01 glink.s
    .fseeko_unlocked 0.01 ccs/lib/libc/fseek.c
    .match_re 0.01 libc/__regexec_std.c
    .strchr 0.01 strchr.s
    .__get_lc_ctype_ptr 0.01 lib/libc/setlocale.c
    .common_open 0.01 /LOCAL/common_file.c
    .free_y 0.01 /lib/libc/malloc_y.c
    .__ftell 0.01 ccs/lib/libc/ftell.c
    .strncmp 0.01 s/lib/libc/strncmp.c
    .strncpy 0.01 strncpy.s
    .atoi 0.01 /ccs/lib/libc/atoi.c
    .strcspn 0.01 s/lib/libc/strcspn.c
    .fgets 0.01 ccs/lib/libc/fgets.c
    ._rec_mutex_unlock 0.01 lib/libc/rec_mutex.c

    Profile: /usr/lib/libc.a[shr_64.o]

    Total % For All Processes (/usr/lib/libc.a[shr_64.o]) = 0.17

    Subroutine % Source
    ========== ====== ======
    .splay 0.02 /lib/libc/malloc_y.c
    .binary_search 0.02 ccs/lib/libs/ntree.c
    .compare_rec_char 0.02 CAL/common_compare.c
    .strncpy 0.02 strncpy.s
    ._doprnt 0.02 cs/lib/libc/doprnt.c
    .time_base_to_time 0.01 /time_base_to_time.c
    .libs_trace 0.01 ib/libs/libs_trace.c
    .leftmost 0.01 /lib/libc/malloc_y.c
    ._findiop 0.01 s/lib/libc/findiop.c
    .__ntree_locate 0.01 ccs/lib/libs/ntree.c
    ._rec_mutex_lock 0.01 lib/libc/rec_mutex.c
    .strpbrk 0.01 s/lib/libc/strpbrk.c
    .getenv 0.01 cs/lib/libc/getenv.c
    .strlen 0.01 strlen.s
    .strchr 0.01 strchr.s

    Profile: /usr/lib/libperfstat.a[shr.o]

    Total % For All Processes (/usr/lib/libperfstat.a[shr.o]) = 0.11

    Subroutine % Source
    ========== ====== ======
    .strncpy 0.04 strncpy.s
    .__get_hashtable_position 0.02 rfstat/dictionnary.c
    .__locked_find_symbol 0.02 rfstat/dictionnary.c
    .__get_disk_description_adapter_and_pvid 0.01 erfstat/disk_utils.c
    .get_lvm_metrics_with_disk_dictionary 0.01 erfstat/disk_utils.c

    Profile: /usr/lib/libpthreads.a[shr_xpg5.o]

    Total % For All Processes (/usr/lib/libpthreads.a[shr_xpg5.o]) = 0.05

    Subroutine % Source
    ========== ====== ======
    .global_unlock_ppc_mp 0.02 pth_locks_ppc_mp.s
    .global_lock_ppc_mp_eh 0.01 th_locks_ppc_mp_eh.s
    .lib_spinlock_lock_ppc_mp_eh 0.01 th_locks_ppc_mp_eh.s
    ._mutex_lock 0.01 pthreads/pth_mutex.c

    Profile: /usr/lib/libct_mss.a[shr.o]

    Total % For All Processes (/usr/lib/libct_mss.a[shr.o]) = 0.02

    Subroutine % Source
    ========== ====== ======
    .bn_addmul_1 0.02 v4/ansic/be32/clic.c
    .sha512 0.01 v4/ansic/be32/clic.c

    Profile: /usr/lib/libpthreads.a[shr.o]

    Total % For All Processes (/usr/lib/libpthreads.a[shr.o]) = 0.02

    Subroutine % Source
    ========== ====== ======
    .global_unlock_ppc_mp 0.01 pth_locks_ppc_mp.s
    .lib_spinlock_unlock_ppc_mp 0.01 pth_locks_ppc_mp.s

    Profile: /usr/bin/ps

    Total % For All Processes (/usr/bin/ps) = 0.02

    Subroutine % Source
    ========== ====== ======
    <0x900000000599200> 0.01
    <0x90000000057ABE0> 0.01

    Profile: /usr/lib/libcrypto.a[libcrypto.so.1.0.0]

    Total % For All Processes (/usr/lib/libcrypto.a[libcrypto.so.1.0.0]) = 0.01

    Subroutine % Source
    ========== ====== ======
    .bn_mul_add_words 0.01 bn_asm.c

    Profile: /usr/lib/libct_skc.a[shr.o]

    Total % For All Processes (/usr/lib/libct_skc.a[shr.o]) = 0.01

    Subroutine % Source
    ========== ====== ======
    .bn_addmul_1 0.01 v4/ansic/be32/clic.c

    Profile: /usr/lib/libC.a[shr.o]

    Total % For All Processes (/usr/lib/libC.a[shr.o]) = 0.01

    Subroutine % Source
    ========== ====== ======
    .ostream::operator<<(const char*) 0.01 ../../out.c

    Profile: /usr/opt/perl5/lib/5.28.1/aix-thread-multi/CORE/libperl.a[libperl.o]

    Total % For All Processes (/usr/opt/perl5/lib/5.28.1/aix-thread-multi/CORE/libperl.a[libperl.o]) = 0.01

    Subroutine % Source
    ========== ====== ======
    .Perl_yylex 0.01 toke.c

    Profile: /usr/lib/libct_rm.a[shr.o]

    Total % For All Processes (/usr/lib/libct_rm.a[shr.o]) = 0.01

    Subroutine % Source
    ========== ====== ======
    .EnumerateResourcesResponseComplete 0.01 api/rm_default_rsp.c

    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<



    ------------------------------
    Roger Andre
    ------------------------------


  • 2.  RE: Problem - High load average, freeze service MUMPS/MSM

    Posted Tue June 21, 2022 09:14 AM
    Hello, today we had another freeze in the system. It was close to 09:15. Logs follow for analysis. Following are MSM/MUMPS logs with about 1050 active connections at the time of the freeze. Does anyone see something different?

    #Active connections
    ^ROGER("66281,35364")="1110|21/06/2022|09:49:24"
    ^ROGER("66281,35374")="1103|21/06/2022|09:49:34"
    ^ROGER("66281,35384")="1098|21/06/2022|09:49:44"
    ^ROGER("66281,35394")="1100|21/06/2022|09:49:54"
    ^ROGER("66281,35404")="1094|21/06/2022|09:50:04"
    ^ROGER("66281,35414")="1091|21/06/2022|09:50:14"
    ^ROGER("66281,35424")="1087|21/06/2022|09:50:24"
    ^ROGER("66281,35434")="1085|21/06/2022|09:50:34"
    ^ROGER("66281,35444")="1081|21/06/2022|09:50:44"
    ^ROGER("66281,35454")="1078|21/06/2022|09:50:54"


    SISMED1:/var/tmp> vmstat -Iwt 2
    kthr memory page faults cpu time
    ----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
    r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
    15 0 0 1090196 71560 146 39 0 0 0 0 9014 394550 66962 19 27 54 0 1.01 50.7 09:55:07
    12 0 0 1090486 70581 322 83 0 0 0 0 11423 517553 83951 23 38 39 0 1.30 64.9 09:55:09
    2 1 0 1090486 70391 89 34 0 0 0 0 10097 462802 77751 21 33 46 0 1.16 58.0 09:55:11
    4 0 0 1090545 70070 116 36 0 0 0 0 10465 481553 74345 21 33 45 0 1.17 58.7 09:55:13
    4 0 0 1090967 69456 89 2 0 0 0 0 10390 470679 76499 21 33 46 0 1.17 58.3 09:55:15
    1 0 0 1090528 69863 18 715 0 0 0 0 2481 78521 13065 47 8 44 1 1.13 56.5 09:55:17
    4 0 0 1090528 69859 0 0 0 0 0 0 99 15008 460 53 3 44 0 1.11 55.7 09:55:19
    2 0 0 1090528 69854 0 0 0 0 0 0 103 20976 564 56 4 40 0 1.20 59.9 09:55:21
    2 0 0 1090528 69850 0 0 0 0 0 0 73 4585 415 52 1 46 0 1.08 53.8 09:55:23
    2 0 0 1090542 69833 0 0 0 0 0 0 52 2112 369 50 1 49 0 1.03 51.4 09:55:25
    2 0 0 1090542 69830 0 1 0 0 0 0 36 1712 331 50 1 49 0 1.02 51.1 09:55:27
    2 0 0 1090556 69813 0 0 0 0 0 0 33 3050 305 50 1 49 0 1.03 51.3 09:55:29
    2 0 0 1090945 69423 0 0 0 0 0 0 41 19132 349 52 3 45 0 1.10 55.0 09:55:31
    2 0 0 1090945 69421 0 0 0 0 0 0 60 2215 446 50 1 49 0 1.03 51.3 09:55:33
    2 0 0 1090945 69418 0 0 0 0 0 0 31 2630 315 50 1 49 0 1.02 51.0 09:55:35
    2 0 0 1090937 69424 0 0 0 0 0 0 43 7827 317 51 1 48 0 1.03 51.7 09:55:37





    SISMED1:/var/tmp> iostat -RDTl 3 1000000000000000000000000000000000000000000000000000000000 > iostat.txt
    Disks: xfers read write queue time
    --------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
    %tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv
    act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_PRD_1 5.0 645.8K 148.7 600.7K 45.1K 144.7 0.3 0.1 2.1 0 0 4.0 0.6 0.4 0.9 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_caa 0.0 1.7K 1.0 1.0K 682.7 0.3 0.2 0.2 0.2 0 0 0.7 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_PRD_2 7.0 532.5K 130.0 532.5K 0.0 130.0 0.4 0.1 4.9 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk13 0.0 25.9K 6.3 0.0 25.9K 0.0 0.0 0.0 0.0 0 0 6.3 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_PRD_3 5.0 537.9K 126.3 516.1K 21.8K 125.7 0.3 0.1 2.1 0 0 0.7 1.1 0.5 1.7 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_PRD_4 5.3 663.6K 136.7 553.0K 110.6K 135.0 0.4 0.1 6.9 0 0 1.7 0.8 0.5 1.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:09:55
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_PRD_1 0.3 91.5K 20.0 36.9K 54.6K 9.0 0.4 0.2 0.5 0 0 11.0 0.8 0.4 1.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_caa 0.0 2.7K 1.3 2.0K 682.7 0.7 0.2 0.2 0.2 0 0 0.7 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_PRD_2 0.7 107.9K 23.7 34.1K 73.7K 8.3 0.4 0.2 1.2 0 0 15.3 0.9 0.5 1.3 0 0 0.0 0.0 0.1 0.0 0.0 0.0 09:55:02
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk13 0.0 15.0K 3.7 0.0 15.0K 0.0 0.0 0.0 0.0 0 0 3.7 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_PRD_3 0.7 136.5K 31.3 41.0K 95.6K 9.7 0.4 0.3 0.5 0 0 21.7 1.0 0.4 1.6 0 0 0.2 0.0 1.1 0.0 0.0 4.3 09:55:02
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_PRD_4 0.3 109.2K 20.3 34.1K 75.1K 8.3 0.4 0.2 0.5 0 0 12.0 0.9 0.5 1.4 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:02
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_PRD_1 0.7 81.9K 18.0 41.0K 41.0K 10.0 0.4 0.2 0.4 0 0 8.0 0.9 0.4 1.4 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_caa 0.0 1.7K 1.0 1.0K 682.7 0.3 0.1 0.1 0.1 0 0 0.7 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_PRD_2 0.3 73.7K 17.7 28.7K 45.1K 7.0 0.4 0.3 0.5 0 0 10.7 1.1 0.5 1.4 0 0 0.0 0.0 0.1 0.0 0.0 0.0 09:55:05
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk13 0.0 19.1K 4.7 0.0 19.1K 0.0 0.0 0.0 0.0 0 0 4.7 0.5 0.2 2.7 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_PRD_3 0.7 116.1K 18.0 49.2K 66.9K 12.0 0.4 0.2 0.4 0 0 6.0 1.0 0.6 1.4 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_PRD_4 0.3 83.3K 17.7 34.1K 49.2K 8.3 0.7 0.3 6.0 0 0 9.3 1.0 0.5 1.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:05
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_PRD_1 1.3 314.0K 75.7 281.3K 32.8K 68.0 0.3 0.1 1.4 0 0 7.7 0.8 0.4 1.7 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_caa 0.0 2.7K 1.3 2.0K 682.7 0.7 2.5 0.2 4.8 0 0 0.7 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_PRD_2 3.0 292.2K 71.3 263.5K 28.7K 64.3 0.4 0.2 5.8 0 0 7.0 1.0 0.6 1.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk13 0.0 12.3K 3.0 0.0 12.3K 0.0 0.0 0.0 0.0 0 0 3.0 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_PRD_3 3.3 327.7K 73.0 269.0K 58.7K 65.7 0.4 0.2 5.6 0 0 7.3 0.9 0.5 1.6 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_PRD_4 3.3 344.1K 77.0 286.7K 57.3K 70.0 0.4 0.1 3.9 0 0 7.0 0.9 0.6 1.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:08
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_PRD_1 1.3 170.7K 39.7 127.0K 43.7K 30.7 0.3 0.2 0.5 0 0 9.0 2.7 0.4 6.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_caa 0.0 1.4K 0.7 1.0K 341.3 0.3 0.2 0.2 0.2 0 0 0.3 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_PRD_2 1.0 165.2K 40.3 124.2K 41.0K 30.3 0.4 0.1 0.6 0 0 10.0 3.4 0.5 6.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk13 1.3 75.1K 18.3 0.0 75.1K 0.0 0.0 0.0 0.0 0 0 18.3 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_PRD_3 1.0 128.3K 31.3 75.1K 53.2K 18.3 0.4 0.3 0.4 0 0 13.0 1.9 0.5 6.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:11
    hdisk_PRD_4 1.7 237.6K 54.0 109.2K 128.3K 26.3 0.4 0.2 0.6 0 0 27.7 2.5 0.4 6.3 0 0 0.1 0.0 4.7 0.0 0.0 1.0 09:55:11
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_PRD_1 1.3 107.9K 26.0 99.7K 8.2K 24.0 0.3 0.1 0.5 0 0 2.0 0.5 0.4 0.6 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_caa 0.0 3.1K 1.7 2.0K 1.0K 0.7 0.2 0.2 0.2 0 0 1.0 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_PRD_2 0.7 120.1K 29.3 120.1K 0.0 29.3 0.4 0.3 5.7 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk13 0.7 17.7K 4.3 0.0 17.7K 0.0 0.0 0.0 0.0 0 0 4.3 0.9 0.2 8.6 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_PRD_3 2.7 154.3K 26.3 101.0K 53.2K 24.3 0.5 0.2 9.0 0 0 2.0 0.6 0.4 0.8 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:14
    hdisk_PRD_4 1.0 139.3K 28.0 92.8K 46.4K 22.0 0.4 0.2 2.2 0 0 6.0 0.6 0.5 0.8 0 0 0.0 0.0 0.1 0.0 0.0 0.0 09:55:14
    hdisk_BKP_9 0.3 1.4K 0.3 0.0 1.4K 0.0 0.0 0.0 0.0 0 0 0.3 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_PRD_1 5.0 573.4K 131.3 47.8K 525.7K 11.7 0.3 0.2 0.4 0 0 119.7 0.8 0.4 11.4 0 0 0.0 0.0 0.8 0.0 0.0 3.0 09:55:17
    hdisk_BKP_10 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_caa 0.0 1.4K 0.7 1.0K 341.3 0.3 0.1 0.1 0.1 0 0 0.3 0.2 0.2 0.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_PRD_2 2.3 379.6K 88.3 41.0K 338.6K 10.0 0.3 0.3 0.4 0 0 78.3 0.9 0.4 11.3 0 0 0.0 0.0 0.1 0.0 0.0 0.0 09:55:17
    hdisk_home 0.3 9.6K 2.3 0.0 9.6K 0.0 0.0 0.0 0.0 0 0 2.3 0.9 0.4 1.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk13 0.3 524.3K 63.3 0.0 524.3K 0.0 0.0 0.0 0.0 0 0 63.3 0.4 0.2 1.4 0 0 0.1 0.0 0.9 0.0 0.0 16.3 09:55:17
    hdisk_PRD_3 3.3 494.3K 107.7 36.9K 457.4K 9.0 0.4 0.3 0.4 0 0 98.7 0.9 0.4 10.8 0 0 0.0 0.0 0.8 0.0 0.0 2.7 09:55:17
    hdisk_BKP_8 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_BKP_7 0.3 1.4K 0.3 0.0 1.4K 0.0 0.0 0.0 0.0 0 0 0.3 0.6 0.6 0.6 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:17
    hdisk_PRD_4 2.0 484.7K 105.7 39.6K 445.1K 9.7 0.4 0.3 1.0 0 0 96.0 1.0 0.4 11.3 0 0 0.2 0.0 2.1 0.0 0.0 17.0 09:55:17
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_PRD_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_4 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_2 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_caa 0.0 3.1K 1.7 2.0K 1.0K 0.7 0.2 0.2 0.2 0 0 1.0 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_6 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_PRD_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_1 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk13 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_PRD_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_3 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_5 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_PRD_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:20
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_PRD_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_caa 0.0 1.4K 0.7 1.0K 341.3 0.3 0.3 0.3 0.3 0 0 0.3 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_PRD_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk13 0.3 21.8K 5.3 0.0 21.8K 0.0 0.0 0.0 0.0 0 0 5.3 0.2 0.2 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_PRD_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_PRD_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:23
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_PRD_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_caa 0.0 2.7K 1.3 2.0K 682.7 0.7 0.2 0.2 0.2 0 0 0.7 0.3 0.3 0.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_PRD_2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_home 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk13 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_PRD_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_PRD_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:26
    hdisk_BKP_9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:29
    hdisk_PRD_1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:29
    hdisk_BKP_10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:29
    hdisk_BKP_4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:55:29





    SISMED1:/var/tmp> tprof -ujeskzl -A -I -N -r default -x sleep 60
    Configuration information
    =========================
    System: AIX 7.1 Node: SISMED1 Machine: 00FB42D74C00
    Tprof command was:
    tprof -ujeskzl -A -I -N -r default -x sleep 60
    Trace command was:
    /usr/bin/trace -ad -M -L 737711308 -T 500000 -j 00A,001,002,003,38F,005,006,134,210,139,5A2,5A5,465,234,5D8, -o default.trc
    Total Samples = 11996
    Traced Time = 60.01s (out of a total execution time of 60.01s)
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

    Process FREQ Total Kernel User Shared Other Java
    ======= ==== ===== ====== ==== ====== ===== ====
    mumsm 1 5997 10 5987 0 0 0
    wait 1 5655 5655 0 0 0 0
    /usr/bin/ps 13 90 60 3 27 0 0
    /usr/es/sbin/cluster/clstat 5 46 35 0 11 0 0
    /usr/sbin/zabbix_agentd: 15 30 22 1 7 0 0
    /usr/sbin/snmpd 1 28 25 0 3 0 0
    /usr/bin/login 4 15 9 0 6 0 0
    /usr/bin/sh 13 14 13 1 0 0 0
    /opt/rsct/bin/rmcd 2 11 7 0 4 0 0
    /usr/sbin/sshd 4 11 5 1 5 0 0
    gil 4 10 10 0 0 0 0
    /usr/sbin/telnetd 5 8 8 0 0 0 0
    lock_rcv 1 7 7 0 0 0 0
    /bin/ksh 5 6 6 0 0 0 0
    /usr/sbin/syslogd 1 5 2 1 2 0 0
    /usr/bin/netstat 3 5 5 0 0 0 0
    /usr/es/sbin/cluster/clinfo 1 5 3 1 1 0 0
    swapper 1 4 4 0 0 0 0
    /usr/bin/w 1 3 3 0 0 0 0
    /usr/bin/wc 3 3 3 0 0 0 0
    /var/opt/tivoli/ep/_jvm/jre/bin/java
    2 3 2 0 1 0 0
    /usr/sbin/rsct/bin/hagsd 1 3 0 0 3 0 0
    /bin/perl 1 3 2 0 1 0 0
    /usr/bin/awk 3 3 3 0 0 0 0
    /usr/java7_64/bin/java 2 3 2 0 1 0 0
    -ksh 3 3 2 0 1 0 0
    mutio 2 3 3 0 0 0 0
    /usr/bin/sort 1 2 1 0 1 0 0
    /usr/bin/sleep 2 2 2 0 0 0 0
    ./msm 2 2 1 0 1 0 0
    DPKP 1 2 2 0 0 0 0
    /usr/sbin/syncd 1 1 1 0 0 0 0
    /usr/bin/iostat 1 1 0 0 1 0 0
    /usr/bin/hostname 1 1 1 0 0 0 0
    /usr/bin/grep 1 1 1 0 0 0 0
    /usr/sbin/cron 1 1 1 0 0 0 0
    /usr/bin/vmstat 1 1 1 0 0 0 0
    /usr/bin/setmaps 1 1 1 0 0 0 0
    /usr/es/sbin/cluster/clstrmgr 1 1 0 0 1 0 0
    /usr/sbin/aso 1 1 1 0 0 0 0
    muctrl 1 1 1 0 0 0 0
    /usr/bin/date 1 1 0 0 1 0 0
    /usr/sbin/bindprocessor 1 1 0 0 1 0 0
    rpc.lockd 1 1 1 0 0 0 0
    /opt/rsct/bin/IBM.MgmtDomainRMd
    1 1 0 0 1 0 0
    ======= ==== ===== ====== ==== ====== ===== ====
    Total 118 11996 5921 5995 80 0 0

    Process PID TID Total Kernel User Shared Other Java
    ======= === === ===== ====== ==== ====== ===== ====
    mumsm 13566366 101908519 5997 10 5987 0 0 0
    wait 131076 131077 5655 5655 0 0 0 0
    /usr/sbin/snmpd 5570750 9043989 28 25 0 3 0 0

    5701654 10616911 10 6 0 4 0 0

    27853170 116523025 10 8 0 2 0 0

    16908568 21037289 9 6 0 3 0 0

    16908564 21037285 9 6 0 3 0 0

    27918726 116326431 9 7 0 2 0 0

    27918720 116326425 9 8 0 1 0 0

    6488312 11468895 8 4 1 3 0 0
    /usr/bin/ps 21496144 116260879 8 4 1 3 0 0
    /usr/bin/ps 19988892 116391957 7 4 0 3 0 0
    /usr/bin/ps 16908558 21037279 7 5 0 2 0 0
    /usr/bin/ps 16187840 115933193 7 4 0 3 0 0
    lock_rcv 2883690 4849827 7 7 0 0 0 0
    /usr/bin/ps 21103086 121634993 7 4 1 2 0 0
    /usr/bin/ps 21496172 116260907 7 5 0 2 0 0
    /usr/bin/ps 21496154 116260889 7 4 0 3 0 0
    /usr/bin/ps 15204564 106758313 7 5 0 2 0 0
    /usr/bin/ps 27722152 114950187 7 6 0 1 0 0
    /usr/bin/ps 15204562 106758311 7 3 1 3 0 0
    /usr/bin/ps 21103060 121634967 7 6 0 1 0 0
    /usr/bin/ps 22938090 110559313 6 6 0 0 0 0

    6684920 11272291 6 5 0 1 0 0
    /usr/sbin/sshd 21496188 116260923 6 2 1 3 0 0
    /usr/bin/ps 21496170 116260905 6 4 0 2 0 0

    16187636 36438113 5 3 1 1 0 0

    4128894 7471333 5 2 1 2 0 0
    swapper 0 3 4 4 0 0 0 0
    /usr/bin/login 26870168 104071203 4 2 0 2 0 0
    /usr/bin/login 26804608 107741261 4 4 0 0 0 0
    gil 1179684 2162755 4 4 0 0 0 0
    /usr/bin/login 19136912 110231567 4 3 0 1 0 0
    /bin/perl 21103062 121634969 3 2 0 1 0 0
    /usr/bin/w 21103044 121634951 3 3 0 0 0 0
    gil 1179684 2097217 3 3 0 0 0 0
    /usr/bin/login 13697434 107216935 3 0 0 3 0 0

    6357204 11534433 3 2 0 1 0 0

    16187848 115933201 3 3 0 0 0 0

    7143560 23920657 3 0 0 3 0 0
    /usr/bin/netstat 16908792 21037257 2 2 0 0 0 0
    /usr/bin/netstat 26804604 107741257 2 2 0 0 0 0
    /bin/ksh 6226110 10747977 2 2 0 0 0 0

    7602186 24707089 2 1 0 1 0 0

    6815952 11599971 2 2 0 0 0 0

    27525488 107937851 2 2 0 0 0 0
    /usr/sbin/sshd 18940338 106299459 2 2 0 0 0 0

    6291456 13303961 2 2 0 0 0 0
    mutio 15991256 121307293 2 2 0 0 0 0
    /usr/bin/sort 27722094 114950385 2 1 0 1 0 0
    /usr/sbin/sshd 18940336 106299457 2 0 0 2 0 0
    gil 1179684 2228293 2 2 0 0 0 0
    DPKP 4653246 9633873 2 2 0 0 0 0
    /usr/bin/sh 21103068 121634975 2 1 1 0 0 0

    21496178 116260913 1 1 0 0 0 0
    /usr/bin/grep 21103054 121634961 1 1 0 0 0 0
    /usr/bin/setmaps 21496142 116260877 1 1 0 0 0 0
    /usr/bin/sh 27722128 114950163 1 1 0 0 0 0
    /usr/bin/sh 27722098 114950389 1 1 0 0 0 0
    /usr/bin/sh 27722108 114950399 1 1 0 0 0 0
    /usr/bin/sh 27722090 114950381 1 1 0 0 0 0
    /usr/bin/sh 16908548 21037269 1 1 0 0 0 0
    /usr/bin/sh 27722138 114950173 1 1 0 0 0 0
    /usr/bin/sh 16908546 21037267 1 1 0 0 0 0
    /usr/bin/sh 21103062 121634969 1 1 0 0 0 0
    /usr/bin/sh 27722122 114950157 1 1 0 0 0 0
    /usr/bin/sh 15204548 106758297 1 1 0 0 0 0
    /usr/bin/sh 27722120 114950155 1 1 0 0 0 0
    /usr/bin/sh 25559450 103612559 1 1 0 0 0 0
    /usr/bin/sleep 17694988 110821625 1 1 0 0 0 0
    /usr/bin/sleep 17695000 110821381 1 1 0 0 0 0
    /usr/bin/date 27722132 114950167 1 0 0 1 0 0
    /usr/bin/awk 18809328 115212293 1 1 0 0 0 0
    /usr/bin/awk 26804598 107741251 1 1 0 0 0 0
    /usr/bin/awk 17694980 110821617 1 1 0 0 0 0

    5701654 12845229 1 1 0 0 0 0

    7012416 14221511 1 0 0 1 0 0

    3866654 12451967 1 0 0 1 0 0

    6291456 8781893 1 0 0 1 0 0
    /usr/sbin/aso 4849814 8192253 1 1 0 0 0 0

    21496164 116260899 1 0 0 1 0 0
    /usr/sbin/cron 15204554 106758303 1 1 0 0 0 0
    /usr/bin/vmstat 24314156 34472159 1 1 0 0 0 0
    /bin/ksh 21496172 116260907 1 1 0 0 0 0
    /usr/sbin/sshd 18809088 115212309 1 1 0 0 0 0
    /usr/bin/iostat 24183068 114294831 1 0 0 1 0 0
    /usr/bin/netstat 21103056 121634963 1 1 0 0 0 0
    -ksh 26804608 107741261 1 0 0 1 0 0

    15204574 106758323 1 1 0 0 0 0

    15204552 106758301 1 1 0 0 0 0

    27722160 106102885 1 1 0 0 0 0

    19988886 116391951 1 0 0 1 0 0
    /bin/ksh 18809324 115212289 1 1 0 0 0 0
    /bin/ksh 27787630 116981781 1 1 0 0 0 0

    15204558 106758307 1 1 0 0 0 0

    16187844 115933197 1 0 0 1 0 0

    21496184 116260919 1 1 0 0 0 0

    27853172 116523027 1 1 0 0 0 0

    13959448 113246405 1 1 0 0 0 0

    27722158 106102883 1 1 0 0 0 0

    27853162 116523017 1 1 0 0 0 0

    27722154 114950189 1 1 0 0 0 0
    /usr/bin/wc 27722098 114950389 1 1 0 0 0 0

    27722110 114950145 1 1 0 0 0 0
    /usr/bin/wc 27853164 116523019 1 1 0 0 0 0

    7602186 20054121 1 1 0 0 0 0

    21954826 109117579 1 1 0 0 0 0
    /bin/ksh 17694996 110821377 1 1 0 0 0 0
    /usr/bin/wc 27918704 116326409 1 1 0 0 0 0
    ./msm 13697434 107216935 1 1 0 0 0 0
    gil 1179684 2031679 1 1 0 0 0 0
    ./msm 26804608 107741261 1 0 0 1 0 0
    muctrl 7995424 28311785 1 1 0 0 0 0
    -ksh 13697434 107216935 1 1 0 0 0 0
    mutio 25821478 113311857 1 1 0 0 0 0
    /usr/sbin/syncd 1638506 5177503 1 1 0 0 0 0
    rpc.lockd 4194446 8454193 1 1 0 0 0 0
    -ksh 18809088 115212309 1 1 0 0 0 0
    ======= === === ===== ====== ==== ====== ===== ====
    Total 11996 5921 5995 80 0 0

    Total Ticks For All Processes (USER) = 5995

    User Process Ticks % Address Bytes
    ============= ===== ====== ======= =====
    mumsm 5987 49.91 100b3674 1234d
    /usr/bin/ps 3 0.03 1000001b0 14768
    /usr/sbin/zabbix_agentd: 1 0.01 100016f8c 1
    /usr/sbin/sshd 1 0.01 10000150 d0080
    /usr/sbin/syslogd 1 0.01 1000001b0 abb0
    /usr/bin/sh 1 0.01 10000100 3eaec
    /usr/es/sbin/cluster/clinfo 1 0.01 10000178 4f402

    Profile: mumsm

    Total Ticks For All Processes (mumsm) = 5987

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    <0x100B36A0> 1061 8.84 100b36a0 1
    <0x100C59AC> 868 7.24 100c59ac 1
    <0x100B3678> 734 6.12 100b3678 1
    <0x100B36D4> 685 5.71 100b36d4 1
    <0x100B3688> 409 3.41 100b3688 1
    <0x100B36D0> 397 3.31 100b36d0 1
    <0x100C5990> 316 2.63 100c5990 1
    <0x100C59C0> 238 1.98 100c59c0 1
    <0x100B368C> 201 1.68 100b368c 1
    <0x100C5950> 196 1.63 100c5950 1
    <0x100B3674> 143 1.19 100b3674 1
    <0x100B36C0> 140 1.17 100b36c0 1
    <0x100B36D8> 126 1.05 100b36d8 1
    <0x100B367C> 122 1.02 100b367c 1
    <0x100C5978> 120 1.00 100c5978 1
    <0x100C5970> 119 0.99 100c5970 1
    <0x100B36C8> 112 0.93 100b36c8 1

    Profile: /usr/bin/ps

    Total Ticks For All Processes (/usr/bin/ps) = 3

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .print_args 1 0.01 /usr/bin/ps bb0 158
    .malloc 1 0.01 glink64.s 13b58 18
    .main 1 0.01 /usr/bin/ps 7830 46ac

    Profile: /usr/sbin/zabbix_agentd:

    Total Ticks For All Processes (/usr/sbin/zabbix_agentd:) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    <0x100016F8C> 1 0.01 100016f8c 1

    Profile: /usr/sbin/sshd

    Total Ticks For All Processes (/usr/sbin/sshd) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .chacha_encrypt_bytes 1 0.01 chacha_private.h 1f864 113c

    Profile: /usr/sbin/syslogd

    Total Ticks For All Processes (/usr/sbin/syslogd) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .iscntrl 1 0.01 glink64.s a9d0 18

    Profile: /usr/bin/sh

    Total Ticks For All Processes (/usr/bin/sh) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .xec_switch 1 0.01 /usr/bin/sh 2b3c0 1f14

    Profile: /usr/es/sbin/cluster/clinfo

    Total Ticks For All Processes (/usr/es/sbin/cluster/clinfo) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .atoi 1 0.01 glink.s c4d8 28

    Total Ticks For All Processes (KERNEL) = 5908

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    h_cede_end_point 5620 46.85 hcalls.s aacf8 8
    .unlock_enable_mem 38 0.32 low.s 930c 1f4
    ovlya_addr_pcs_glue_xon2 29 0.24 vmvcs.s 755ac 8c
    .enable 15 0.13 misc.s 1cb590 70
    .simple_unlock_mem 13 0.11 low.s 9918 1e8
    .v_lookup_mpss 11 0.09 ../../../../../src/bos/kernel/vmm/v_lookup.c 17f700 420
    .hkeyset_restore 10 0.08 low.s 14c24 24
    .simple_lock 9 0.08 low.s 9500 400
    sc_msr_2_point 8 0.07 low.s 3840 24
    svc_rfid_ret32 7 0.06 low.s 3bec 2a0
    .v_lookup 7 0.06 ../../../../../src/bos/kernel/vmm/v_lookup.c 181ae0 3a0
    .fill_procentry64 7 0.06 ../../../../../src/bos/kernel/proc/getproc.c 465480 960
    hypervisor_call_asm_end_point 7 0.06 hcalls.s ab2b8 18
    .umem_move 5 0.04 low.s b600 200
    .hkeyset_add 5 0.04 low.s 14c00 24
    svc_rfid_ret64 4 0.03 low.s 3b9c 14
    .disable_lock 4 0.03 low.s 9004 2fc
    .v_lookup_swpft 4 0.03 ../../../../../src/bos/kernel/vmm/v_lookup.c 17fdc0 280
    .v_insscb 3 0.03 ../../../../../src/bos/kernel/vmm/v_scblist.c 2d9b40 440
    .trchook64 3 0.03 trchka64.s b1fb0 450
    .nlcLookup 3 0.03 ../../../../../src/bos/kernel/lfs/nlc.c 790760 400
    .kernel_add_gate_cstack 2 0.02 low.s 14f20 60
    hk_update_ukeys_accr_point 2 0.02 64/skeys.s a8840 30
    .v_deque_nfr 2 0.02 ../../../../../src/bos/kernel/vmm/v_freelist.c 191260 420
    .vm_handle 2 0.02 ../../../../../src/bos/kernel/vmm/vmmisc64.c 3b840 e80
    .kern_geth 2 0.02 ../../../../../src/bos/kernel/vmm/vmadsputil.c 11c480 4c0
    .fetch_and_add 2 0.02 low.s 9b00 80
    .crexport 2 0.02 ../../../../../../src/bos/kernel/s/auth/cred.c 4b3c0 280
    .v_inspft 2 0.02 ../../../../../src/bos/kernel/vmm/v_lists.c 25ae60 280
    .v_update_frame_stats 2 0.02 ../../../../../src/bos/kernel/vmm/v_lists.c 25b0e0 920
    .mtrchook2 2 0.02 low.s 10204 38
    .v_scan_compute_weights 2 0.02 ../../../../../src/bos/kernel/vmm/vmscan.c 2e4f80 3c0
    .j2_lookup 2 0.02 ../../../../../src/bos/kernel/j2/j2_lookup.c 3c4f80 580
    .v_wpagein 2 0.02 ../../../../../src/bos/kernel/vmm/v_getsubs1.c 183c00 1320
    .getthrds 2 0.02 ../../../../../src/bos/kernel/proc/getproc.c 46c9e0 b60
    .copyin 1 0.01 ../../../../../src/bos/kernel/vmm/userio.c a7a40 220
    .VM_GETKEY 1 0.01 ../../../../../src/bos/kernel/vmm/vmmisc64.c 3c800 40
    ._hkeyset_restore_userkeys 1 0.01 64/skeys.s a8870 10
    .drw_lock_read 1 0.01 low.s 15d20 120
    ._ptrgl 1 0.01 low.s 14d00 24
    ._thread_unlock_common 1 0.01 ../../../../../src/bos/kernel/proc/sleep3.c be1c0 4c0
    .clock 1 0.01 ../../../../../src/bos/kernel/proc/clock.c c9220 580
    .xmattach 1 0.01 ../../../../../src/bos/kernel/vmm/xmem.c 11b0c0 320
    .hkeyset_replace 1 0.01 low.s 14c48 24
    .vm_add_xmemcnt 1 0.01 ../../../../../src/bos/kernel/vmm/v_segsubs.c 1226a0 200
    .vm_att 1 0.01 low.s b484 7c
    .lock_done_mem 1 0.01 low.s be48 178
    .v_ff_bitmap_upd 1 0.01 ../../../../../src/bos/kernel/vmm/v_freelist.c 18fd00 300
    .v_deque_nfr_ff_bitmap 1 0.01 ../../../../../src/bos/kernel/vmm/v_freelist.c 1901e0 160
    .lock_read 1 0.01 low.s bd20 120
    .get_64bit_rlimit_u 1 0.01 ../../../../../src/bos/kernel/proc/resource_pn.c 1b6640 e0
    .vm_det 1 0.01 low.s b504 7c
    sc_msr_1_point 1 0.01 low.s 3a08 c
    .cdev_rdwr 1 0.01 ../../../../../src/bos/kernel/specfs/cdev_subr.c 8a9180 4a0
    .v_psize_freeok 1 0.01 ../../../../../src/bos/kernel/vmm/vmpsize.c 5cc60 1a0
    .v_get_validated_bsidx 1 0.01 ../../../../../src/bos/kernel/vmm/v_scbsubs.c 6b4a0 1a0
    .v_sort_wsidlist 1 0.01 ../../../../../src/bos/kernel/vmm/v_relsubs.c 2bbd60 560
    .v_relframe 1 0.01 ../../../../../src/bos/kernel/vmm/v_relsubs.c 2c10c0 e00
    .v_reclaim 1 0.01 ../../../../../src/bos/kernel/vmm/v_getsubs.c 2d1160 1220
    .v_validate_sidx 1 0.01 ../../../../../src/bos/kernel/vmm/v_scbsubs.c 6b640 12c0
    .v_descoreboard 1 0.01 ../../../../../src/bos/kernel/vmm/v_mpsubs.c 6f2e0 240
    .v_scan_end 1 0.01 ../../../../../src/bos/kernel/vmm/vmscan.c 2e6c00 100
    .pile_object_wanted 1 0.01 ../../../../../src/bos/kernel/j2/j2_inode.c 3108a0 40
    .iActivate 1 0.01 ../../../../../src/bos/kernel/j2/j2_inode.c 315280 500
    .bmAssign 1 0.01 ../../../../../src/bos/kernel/j2/j2_bufmgr.c 32d520 c40
    .check_free 1 0.01 ../../../../../src/bos/kernel/vmm/vmxmdbg.c 373680 3a0
    .crxref 1 0.01 ../../../../../../src/bos/kernel/s/auth/cred.c 4bce0 e0
    .j2_seek 1 0.01 ../../../../../src/bos/kernel/j2/j2_seek.c 3d2180 80
    .validfault 1 0.01 ../../../../../src/bos/kernel/vmm/v_getsubs64.c 403240 1a0
    .waitproc_find_run_queue 1 0.01 ../../../../../src/bos/kernel/proc/dispatch.c 7fb60 960
    .getptr 1 0.01 ../../../../../src/bos/kernel/proc/getproc.c 46a020 1a0
    .segattach 1 0.01 ../../../../../src/bos/kernel/proc/getproc.c 46af80 3a0
    .waitproc 1 0.01 ../../../../../src/bos/kernel/proc/dispatch.c 86f60 9c0
    .rtfree_nolock 1 0.01 ../../../../../src/bos/kernel/net/route.c 501100 1
    .rtalloc1_nolock_gr 1 0.01 ../../../../../src/bos/kernel/net/route.c 501500 780
    .m_copydata 1 0.01 ../../../../../src/bos/kernel/uipc/mbuf.c 52e800 1
    .uipc_usrreq 1 0.01 ../../../../../src/bos/kernel/uipc/usrreq.c 54e480 1
    .soclose2 1 0.01 ../../../../../src/bos/kernel/uipc/socket.c 57a700 e00
    .erecvit 1 0.01 ../../../../../src/bos/kernel/uipc/syscalls.c 592c00 1580
    .sbdrop 1 0.01 ../../../../../src/bos/kernel/uipc/socket2.c 5a8580 1
    .closex 1 0.01 ../../../../../src/bos/kernel/lfs/close.c 605020 3e0
    .lookuppn 1 0.01 ../../../../../src/bos/kernel/lfs/lookuppn.c 606140 10c0
    .xmalloc 1 0.01 ../../../../../src/bos/kernel/alloc/xmalloc.c 62eec0 9c0
    .vnop_rele 1 0.01 ../../../../../src/bos/kernel/lfs/vnops.c 69dce0 180
    .vnop_ioctl 1 0.01 ../../../../../src/bos/kernel/lfs/vnops.c 6a0600 200
    .vnop_getattr_flags 1 0.01 ../../../../../src/bos/kernel/lfs/vnops.c 6a0d60 1a0
    ._kvmgetinfo 1 0.01 ../../../../../src/bos/kernel/vmm/vmgetinfo.c 728c20 840
    .vminfo64to32 1 0.01 ../../../../../src/bos/kernel/vmm/vmgetinfo.c 72d820 340
    .copyinstr 1 0.01 ../../../../../src/bos/kernel/vmm/userio.c a7600 220
    .uinfox_ref 1 0.01 ../../../../../src/bos/kernel/proc/usrinfo.c 7fd4e0 a0
    .ld_resolve1 1 0.01 ../../../../../src/bos/kernel/ldr/ld_symbols.c 86a520 1920
    .ld_hash 1 0.01 ../../../../../src/bos/kernel/ldr/ld_symbols.c 86f3e0 c0

    Millicode Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .memset_overlay 6 0.05 low.s e008 5f8
    .memset64_overlay 2 0.02 low.s e808 5f8
    .strlen_overlay 1 0.01 low.s e600 200
    .strcpy_overlay 1 0.01 low.s fc00 200
    .strcmp_overlay 1 0.01 low.s dc00 200

    Total Ticks For All Processes (KEX) = 13

    Kernel Ext Ticks % Address Bytes
    ========== ===== ====== ======= =====
    /usr/lib/drivers/netinet 5 0.04 5cd0200 ef8a0
    UnknownBinary 3 0.03 f1000000c03e7094 1
    /usr/lib/drivers/eth_demux 3 0.03 5ca0200 3e30
    /usr/lib/drivers/if_en 1 0.01 5fa0200 7d58
    /usr/lib/drivers/vioentdd 1 0.01 5c50300 24a48

    Profile: /usr/lib/drivers/netinet

    Total Ticks For All Processes (/usr/lib/drivers/netinet) = 5

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .tcp_fasttimo 2 0.02 /usr/lib/drivers/netinet df4e0 420
    .tcp_slowtimo 2 0.02 /usr/lib/drivers/netinet dee40 668
    .tcp_input0 1 0.01 /usr/lib/drivers/netinet bd700 a7dc

    Profile: UnknownBinary

    Total Ticks For All Processes (UnknownBinary) = 3

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    <0xF1000000C03E7094> 3 0.03 f1000000c03e7094 1

    Profile: /usr/lib/drivers/eth_demux

    Total Ticks For All Processes (/usr/lib/drivers/eth_demux) = 3

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .eth_std_receive 2 0.02 /usr/lib/drivers/eth_demux 2800 4bc
    .eth_receive 1 0.01 /usr/lib/drivers/eth_demux 860 3ac

    Profile: /usr/lib/drivers/if_en

    Total Ticks For All Processes (/usr/lib/drivers/if_en) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .en_output 1 0.01 /usr/lib/drivers/if_en 2da0 1898

    Profile: /usr/lib/drivers/vioentdd

    Total Ticks For All Processes (/usr/lib/drivers/vioentdd) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .vioent_receive 1 0.01 ../../../../../src/bos/kernext/vioent/vioent_receive.c 18b00 2020

    Total Ticks For All Processes (SH-LIBs) = 80

    Shared Object Ticks % Address Bytes
    ============= ===== ====== ======= =====
    /usr/lib/libc.a[shr_64.o] 28 0.23 900000000000a00 359438
    /usr/lib/libc.a[shr.o] 25 0.21 d0100280 361448
    /usr/bin/ps 6 0.05 900000000579a60 218dd
    /usr/lib/libpthreads.a[shr_xpg5.o] 5 0.04 d052f180 32ced
    /usr/lib/libcrypto.a[libcrypto.so.1.0.0] 3 0.03 d414c350 15ec1e
    /usr/lib/libpthreads.a[shr.o] 3 0.03 d0c39180 2d10d
    /usr/lib/libi18n.a[shr.o] 2 0.02 d057e280 b180
    /usr/lib/libsnmppriv_ne.a[shr.o] 2 0.02 d0c7c280 7ddac
    /usr/lib/libct_mss.a[shr.o] 2 0.02 d469a280 8bc50
    /usr/lib/libperfstat.a[shr_64.o] 1 0.01 900000000911b00 67273
    /usr/sbin/zabbix_agentd: 1 0.01 900000000579ac0 1
    /usr/java7_64/jre/lib/ppc64/compressedrefs/libj9jit26.so 1 0.01 900000000cea280 8ca0f4
    Millicode routines 1 0.01 0 3200

    Profile: /usr/lib/libc.a[shr_64.o]

    Total Ticks For All Processes (/usr/lib/libc.a[shr_64.o]) = 28

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .strchr 3 0.03 strchr.s 1c000 180
    .compare_rec_char 3 0.03 ../../../../../../../src/bos/usr/lib/security/LOCAL/common_compare.c 1993e0 220
    ._doprnt 3 0.03 ../../../../../../../src/bos/usr/ccs/lib/libc/doprnt.c 5a80 81a0
    .compare_rec_int 3 0.03 ../../../../../../../src/bos/usr/lib/security/LOCAL/common_compare.c 199240 1a0
    .binary_search 2 0.02 ../../../../../../../src/bos/usr/ccs/lib/libs/ntree.c 1067c0 2a0
    .iscntrl 2 0.02 ../../../../../../../src/bos/usr/ccs/lib/libc/iscntrl.c 279500 100
    .free_y 2 0.02 ../../../../../../../src/bos/usr/ccs/lib/libc/malloc_y.c 40cc0 9a0
    .colon_search 1 0.01 ../../../../../../../src/bos/usr/lib/security/LOCAL/file_colon.c 1ad300 1120
    .ntree_search 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libs/ntree.c 105b20 340
    ._select 1 0.01 glink64.s 156d28 1
    .malloc_y 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/malloc_y.c 42780 840
    .read_colon_rec 1 0.01 ../../../../../../../src/bos/usr/lib/security/LOCAL/common_file.c 190840 720
    .__ftell 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/ftell.c 3db00 3a0
    .strncpy 1 0.01 strncpy.s 3c080 128
    .malloc_common@AF104_87 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/malloc_common.c 2aa20 1e0
    .malloc 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/malloc_common.c 2a720 120
    .__ntree_locate 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libs/ntree.c 105e60 620

    Profile: /usr/lib/libc.a[shr.o]

    Total Ticks For All Processes (/usr/lib/libc.a[shr.o]) = 25

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .strpbrk 3 0.03 ../../../../../../../src/bos/usr/ccs/lib/libc/strpbrk.c 23500 100
    .strncpy 2 0.02 strncpy.s 23c00 140
    .fgets 2 0.02 ../../../../../../../src/bos/usr/ccs/lib/libc/fgets.c 1bd80 380
    .strsep 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libirs/strsep.c 14f580 100
    .sv_next 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libirs/lcl_sv.c 1326e0 2c0
    .ntree_walk 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libs/ntree.c 1134c0 200
    .string 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/doscan.c f7f60 560
    .time_base_to_time 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/POWER/time_base_to_time.c ef680 400
    .read_wall_time 1 0.01 read_real_time.s ef580 100
    .__nl_langinfo_std 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/__nl_langinfo_std.c 223940 c0
    ._q_cvtl 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/_q_cvtl.c a0000 120
    .colon_search 1 0.01 ../../../../../../../src/bos/usr/lib/security/LOCAL/file_colon.c 1b6460 1080
    .sigemptyset 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/sigops.c 7cd00 80
    .mbtowc 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/mbtowc.c 39300 180
    ._doprnt 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/doprnt.c 2e880 7fc0
    .splay 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/malloc_y.c 27a80 520
    .__ftell 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/ftell.c 25880 3a0
    .compare_rec_char 1 0.01 ../../../../../../../src/bos/usr/lib/security/LOCAL/common_compare.c 1a3060 220
    .atoi 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/atoi.c 23600 600
    .__fd_select 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/iosl.c 161ac0 220
    .fcntl 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libc/fcntl.c 1f3c0 c0

    Profile: /usr/bin/ps

    Total Ticks For All Processes (/usr/bin/ps) = 6

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    <0x90000000057ABE0> 2 0.02 90000000057abe0 1
    <0x90000000059B33C> 1 0.01 90000000059b33c 1
    <0x90000000057ABE8> 1 0.01 90000000057abe8 1
    <0x900000000579A60> 1 0.01 900000000579a60 1
    <0x90000000059B304> 1 0.01 90000000059b304 1

    Profile: /usr/lib/libpthreads.a[shr_xpg5.o]

    Total Ticks For All Processes (/usr/lib/libpthreads.a[shr_xpg5.o]) = 5

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .global_unlock_ppc_mp 2 0.02 pth_locks_ppc_mp.s 2e824 dc
    .global_lock_ppc_mp_eh 2 0.02 pth_locks_ppc_mp_eh.s 2e724 dc
    .thread_tsleep 1 0.01 glink.s 17a50 28

    Profile: /usr/lib/libcrypto.a[libcrypto.so.1.0.0]

    Total Ticks For All Processes (/usr/lib/libcrypto.a[libcrypto.so.1.0.0]) = 3

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .bn_mul_mont_fpu64 3 0.03 ppc64-mont.s 1e2b0 f00

    Profile: /usr/lib/libpthreads.a[shr.o]

    Total Ticks For All Processes (/usr/lib/libpthreads.a[shr.o]) = 3

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .pthread_cleanup_pop 2 0.02 ../../../../../../../../src/bos/usr/ccs/lib/libpthreads/pth_pthread.c eb00 80
    .pthread_cleanup_push 1 0.01 ../../../../../../../../src/bos/usr/ccs/lib/libpthreads/pth_pthread.c eb80 80

    Profile: /usr/lib/libi18n.a[shr.o]

    Total Ticks For All Processes (/usr/lib/libi18n.a[shr.o]) = 2

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .__wctomb_iso1 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libi18n/__wctomb_iso1.c 3100 80
    .__mbtowc_iso1 1 0.01 ../../../../../../../src/bos/usr/ccs/lib/libi18n/__mbtowc_iso1.c 2780 180

    Profile: /usr/lib/libsnmppriv_ne.a[shr.o]

    Total Ticks For All Processes (/usr/lib/libsnmppriv_ne.a[shr.o]) = 2

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .asn_decode_length 1 0.01 ../../../../../../src/tcpip/usr/lib/libsnmppriv_ne/snmp_asn1.c 39780 120
    .fstatx 1 0.01 glink.s 46d0 1

    Profile: /usr/lib/libct_mss.a[shr.o]

    Total Ticks For All Processes (/usr/lib/libct_mss.a[shr.o]) = 2

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .sha512 2 0.02 /afs/apd.pok.ibm.com/u/myungbae/sandboxes/base_mut/src/rsct/crypto/clicv4/ansic/be32/clic.c 4190 36dc

    Profile: /usr/lib/libperfstat.a[shr_64.o]

    Total Ticks For All Processes (/usr/lib/libperfstat.a[shr_64.o]) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .strncpy 1 0.01 strncpy.s 2300 140

    Profile: /usr/sbin/zabbix_agentd:

    Total Ticks For All Processes (/usr/sbin/zabbix_agentd:) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    <0x900000000579AC0> 1 0.01 900000000579ac0 1

    Profile: /usr/java7_64/jre/lib/ppc64/compressedrefs/libj9jit26.so

    Total Ticks For All Processes (/usr/java7_64/jre/lib/ppc64/compressedrefs/libj9jit26.so) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .samplerThreadProc(void*) 1 0.01 HookedByTheJit.cpp 175e00 f40

    Profile: Millicode routines

    Total Ticks For All Processes (Millicode routines) = 1

    Subroutine Ticks % Source Address Bytes
    ========== ===== ====== ====== ======= =====
    .mull 1 0.01 low.s 3180 80

    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<



    ------------------------------
    Roger Andre
    ------------------------------