AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
Back to discussions
Expand all | Collapse all

hdisk0 and hdisk1 100% utilization and system went down slow

  • 1.  hdisk0 and hdisk1 100% utilization and system went down slow

    Posted Tue January 25, 2011 07:41 AM

    Originally posted by: sajid1


    Hi,

    Whenever the backup(using dsmagent as below output) is started the database server hdisk0 and hdisk1 shows 100% utilization and the all application servers and database server went damn slowly even the application login page does not open. A single AIX command ls takes long time to give output.
    For your information where the oracle home and data files are located on hdisk3 and hdisk4. Find below.
    $ lspv -l hdisk0
    hdisk0:
    LV NAME LPs PPs DISTRIBUTION MOUNT POINT
    hd10opt 1 1 00..00..01..00..00 /opt
    hd1 4 4 00..00..04..00..00 /home
    lg_dumplv 8 8 00..08..00..00..00 N/A
    hd5 1 1 01..00..00..00..00 N/A
    hd8 1 1 00..00..01..00..00 N/A
    hd6 77 77 00..77..00..00..00 N/A
    hd2 20 20 00..00..20..00..00 /usr
    hd4 4 4 00..00..04..00..00 /
    hd3 24 24 00..00..24..00..00 /tmp
    hd9var 8 8 00..00..08..00..00 /var
    $ lspv -l hdisk1
    hdisk1:
    LV NAME LPs PPs DISTRIBUTION MOUNT POINT
    hd10opt 1 1 00..00..01..00..00 /opt
    hd1 4 4 00..00..04..00..00 /home
    lv02 7 7 00..07..00..00..00 /mkcd/cd_images
    loglv01 1 1 00..01..00..00..00 N/A
    hd5 1 1 01..00..00..00..00 N/A
    hd8 1 1 00..00..01..00..00 N/A
    hd6 77 77 00..77..00..00..00 N/A
    hd2 20 20 00..00..20..00..00 /usr
    hd4 4 4 00..00..04..00..00 /
    hd3 24 24 00..00..24..00..00 /tmp
    hd9var 8 8 00..00..08..00..00 /var
    Topas Monitor for host: devdb EVENTS/QUEUES FILE/TTY
    Tue Jan 25 15:20:51 2011 Interval: 2 Cswitch 10116 Readch 0.1G
    Syscall 11853 Writech 82292
    Kernel 8.3 |### | Reads 2062 Rawin 0
    User 3.2 |## | Writes 162 Ttyout 234
    Wait 8.0 |### | Forks 0 Igets 0
    Idle 80.5 |####################### | Execs 0 Namei 60
    Physc = 0.61 %Entc= 12.2 Runqueue 2.5 Dirblk 0
    Waitqueue 19.0
    Network KBPS I-Pack O-Pack KB-In KB-Out
    en3 55.0K 4521.5 2618.4 241.8 54.8K PAGING MEMORY
    lo0 1.3 2.0 2.0 0.7 0.7 Faults 1145 Real,MB 16384
    en2 0.0 0.0 0.0 0.0 0.0 Steals 15392 % Comp 45.9
    PgspIn 676 % Noncomp 54.5
    Disk Busy% KBPS TPS KB-Read KB-Writ PgspOut 153 % Client 54.5
    dac0 0.0 54.8K 184.3 54.8K 0.0K PageIn 15320
    hdisk4 19.4 54.8K 184.3 54.8K 0.0K PageOut 169 PAGING SPACE
    hdisk0 100.3 2030.0 436.6 1418.6 611.4K Sios 15489 Size,MB 19712
    hdisk1 100.3 2022.0 429.1 1408.6 613.4K % Used 30.7
    NFS (calls/sec) % Free 69.2
    Name PID CPU% PgSp Owner ServerV2 0
    dsmagent 1859616 3.5 5.9 root ClientV2 0 Press:
    lrud 16392 1.5 1.2 root ServerV3 0 "h" for help
    xmwlm 315546 0.7 0.9 root ClientV3 0 "q" to quit
    topas 3928238 0.6 5.7 oracle
    #AIX-Forum


  • 2.  Re: hdisk0 and hdisk1 100% utilization and system went down slow

    Posted Tue January 25, 2011 07:43 AM

    Originally posted by: sajid1


    Output of vmstat and iostat
    $ vmstat 2 5

    System configuration: lcpu=10 mem=16384MB ent=5.00

    kthr memory page faults cpu
    -----------
    ------------
    r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
    2 10 3017550 20078 0 668 34 903 1543 0 4785 20875 11831 5 6 85 4 0.60
    12.0
    2 10 3017061 20093 0 694 0 452 516 0 4458 14212 11065 3 5 86 5 0.46
    9.1
    1 12 3017082 20087 0 537 172 11240 23362 0 4400 21575 9288 3 11 81 5 0.7
    3 14.5
    4 10 3017112 20329 0 657 75 17779 70241 0 5382 19367 14131 5 14 50 31 1.
    07 21.4
    10 9 3023685 20084 0 578 136 16137 72370 0 4844 14049 13183 5 12 65 18 0.
    91 18.2
    $ iostat 2 3

    System configuration: lcpu=10 drives=10 ent=5.00 paths=3 vdisks=0

    tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
    0.0 33.1 6.3 6.5 78.2 8.9 0.7 13.3

    Disks: % tm_act Kbps tps Kb_read Kb_wrtn
    dac0 0.0 33782.0 437.8 70352 16
    dac0utm 0.0 0.0 0.0 0 0
    hdisk1 98.9 2434.9 497.4 816 4256
    hdisk0 98.9 2413.8 493.5 732 4296
    hdisk2 0.0 0.0 0.0 0 0
    dac1 0.0 93559.3 926.5 194740 144
    dac1utm 0.0 0.0 0.0 0 0
    hdisk3 98.9 93559.3 926.5 194740 144
    hdisk4 75.4 33782.0 437.8 70352 16
    hdisk5 0.0 0.0 0.0 0 0

    tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
    0.0 404.9 8.2 9.4 63.8 18.6 0.9 18.8

    Disks: % tm_act Kbps tps Kb_read Kb_wrtn
    dac0 0.0 27934.9 415.6 60088 0
    dac0utm 0.0 0.0 0.0 0 0
    hdisk1 100.4 1844.7 428.6 2512 1456
    hdisk0 100.0 1842.9 429.1 2548 1416
    hdisk2 0.0 0.0 0.0 0 0
    dac1 0.0 96087.4 939.1 206588 96
    dac1utm 0.0 0.0 0.0 0 0
    hdisk3 100.4 96087.4 939.1 206588 96
    hdisk4 67.4 27934.9 415.6 60088 0
    hdisk5 0.0 0.0 0.0 0 0

    tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
    0.0 433.8 7.4 8.7 53.6 30.3 0.9 17.4

    Disks: % tm_act Kbps tps Kb_read Kb_wrtn
    dac0 0.0 3510.0 65.7 6808 240
    dac0utm 0.0 0.0 0.0 0 0
    hdisk1 99.6 1402.4 348.6 2616 200
    hdisk0 99.6 1478.1 367.0 2768 200
    hdisk2 0.0 0.0 0.0 0 0
    dac1 0.0 88119.5 946.2 176832 112
    dac1utm 0.0 0.0 0.0 0 0
    hdisk3 99.6 88119.5 946.2 176832 112
    hdisk4 11.5 3510.0 65.7 6808 240
    hdisk5 0.0 0.0 0.0 0 0
    #AIX-Forum


  • 3.  Re: hdisk0 and hdisk1 100% utilization and system went down slow

    Posted Sun January 30, 2011 10:22 AM

    Originally posted by: sajid1


    Kindly Answer me.
    #AIX-Forum


Related Content