AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
  • 1.  Poor performance on disk

    Posted Wed October 28, 2009 09:51 AM

    Originally posted by: Troispoils


    Hi all,

    My concern is about poor disk performance on AIX box.
    This box is binded to VIO which is connected to a NetApp FAS3020 through SAN.

    All qdepth have been set to 32.

    In detail, i copy paste an extract of filemon. This filemon run during 15 seconds.
    I'm very surprised about the max read time (around 5 seconds !).

    What do you think about that ?

    Many Thx

    Jacques


    Detailed Logical Volume Stats (512 byte blocks)


    VOLUME: /dev/bt35ede1-5 description: /ede1x/prd
    reads: 1647 (0 errs)
    read sizes (blks): avg 32.4 min 8 max 256 sdev 40.8
    read times (msec): avg 15.011 min 0.219 max 5888.124 sdev 195.066
    read sequences: 1082
    read seq. lengths: avg 49.3 min 8 max 816 sdev 84.2
    writes: 65 (0 errs)
    write sizes (blks): avg 16.2 min 8 max 32 sdev 7.3
    write times (msec): avg 1.352 min 0.460 max 6.044 sdev 1.351
    write sequences: 65
    write seq. lengths: avg 16.2 min 8 max 32 sdev 7.3
    seeks: 1147 (67.0%)
    seek dist (blks): init 127218864,
    avg 9424983.9 min 8 max 119147216 sdev 18926868.5
    time to next req(msec): avg 8.493 min 0.000 max 149.963 sdev 19.151
    throughput: 1871.4 KB/sec
    utilization: 0.96

    VOLUME: /dev/bt35ehr1-A description: /ehr1x/prd
    reads: 113 (0 errs)
    read sizes (blks): avg 30.7 min 8 max 128 sdev 31.4
    read times (msec): avg 14.012 min 0.451 max 236.869 sdev 27.489
    read sequences: 99
    read seq. lengths: avg 35.0 min 8 max 288 sdev 47.1
    writes: 29 (0 errs)
    write sizes (blks): avg 15.4 min 8 max 56 sdev 11.7
    write times (msec): avg 20.791 min 0.445 max 535.117 sdev 100.871
    write sequences: 29
    write seq. lengths: avg 15.4 min 8 max 56 sdev 11.7
    seeks: 128 (90.1%)
    seek dist (blks): init 220329584,
    avg 27816990.8 min 8 max 106576304 sdev 29200314.5
    time to next req(msec): avg 94.933 min 0.001 max 1642.751 sdev 256.921
    throughput: 134.5 KB/sec
    utilization: 0.90
    #AIX-Forum


  • 2.  Re: Poor performance on disk

    Posted Wed October 28, 2009 11:12 AM

    Originally posted by: MarkTaylor


    Be careful .. your avg and sdev times look ok .. you are doing some pretty hefty seeks btw in that output and that is prob why your disk is showing heavy utilisation .. do you in fact have an issue or are you just looking at stats trying to find one ? I would start by benchmarking the disk in question with some known factors i.e. dd throughput and ndisk and get a picture of what the disk can do and take it from there .. queue_depth on the virtual disk on the client can be changed to match the queue_depth of the disk on the VIOServer forget which TL enabled this, 5.3 TL3 ??) make sure your DMA and cmd elems is set correctly on the FC cards on the VIOSserver too .. Also, bear in mind the number of virtual disks you have hanging off of a vhost --> vscsi .. with a queue_depth of 3 its 80 max, with a queue_depth of 32 you reduce the number of disks you can have to about 8'ish ..

    HTH
    Mark Taylor
    #AIX-Forum


  • 3.  Re: Poor performance on disk

    Posted Wed October 28, 2009 11:36 AM

    Originally posted by: Troispoils


    Thanks Mark,

    Actually, users complain about poor performance on OAP that's why after checking CPU/RAM and storage itself, i drill down to filemon analysis.

    According to the NetApp stats, the avg read time during this period is around 10ms ( which matches with AIX avg time). I really don't understand why max read time can exceed few seconds. i've also try a new run which shows 25 seconds.... Should I trust the measure ? If yes, how to explain this huge value...

    Jacques
    #AIX-Forum