Global Storage

Global Storage Forum

Connect, collaborate, and stay informed with insights from across Storage

 View Only

  • 1.  Help needed: IBM FlashSystem performance issues after firmware upgrade?

    Posted 07/24/26 08:39 AM
    We recently encountered an unexpected issue with our IBM FlashSystem storage performance after upgrading to the latest firmware and driver versions. Despite reverting to the previous stable release, the problem persists intermittently, causing occasional latency spikes and occasional data transfer failures. The setup involves two FlashSystem A9000R arrays connected via Fibre Channel to IBM Power servers running AIX, with IBM Spectrum Virtualize for flash acting as the storage virtualization layer.
    Key observations:
    • Latency spikes up to 200ms during peak workloads, despite normal CPU and memory utilization.
    • Random I/O read/write failures logged in both host and storage event logs.
    • HBA firmware, multipath daemon, and AIX LPAR settings remain unchanged since last stable operation.
    We’d greatly appreciate any insights or similar experiences from the community regarding troubleshooting steps for post-firmware performance anomalies on IBM FlashSystem storage. Specific questions:
    • Have you encountered similar latency or I/O failure patterns after firmware upgrades?
    • Are there recommended diagnostic utilities beyond the standard flashsystemcli and AIX errpt?
    • Could misaligned multipath settings or path failures be a contributing factor despite unchanged configurations?
    Thank you in advance for any pointers or suggestions to help us stabilize our environment.


  • 2.  RE: Help needed: IBM FlashSystem performance issues after firmware upgrade?

    Posted 07/24/26 11:12 AM
    Edited by Nezih Boyacioglu 07/24/26 12:06 PM
    Hi, 
     
    As you know, it's not possible to solve your issue by saying something or entering certain parameters. A performance bottleneck could be anywhere. You can check our latest IBM Redbooks publication for AIX performance tuning, SAN connectivity, and FlashSystem best practices.  The Definitive Guide to IBM Storage FlashSystem and AIX Integration


    To resolve performance issues, you need a performance monitoring tool to provide the required logs to the IBM Support.

     
    ------------------------------
    Nezih Boyacioglu
    IBM Community Hero
    https://nezihboyacioglu.com
    ------------------------------



  • 3.  RE: Help needed: IBM FlashSystem performance issues after firmware upgrade?
    Best Answer

    Posted 07/27/26 03:52 AM

    Goodday

    What is the version of AIX?

    Are you using AIX MPIO drivers are you using? ( i.e. AIX_AAPCM or MPIO_XIVCTRL)

    What is the firmware of A9000R? (IBM dropped support for this sometime ago)

    Which workloads/application is this server running?

    The bottleneck can be caused by a combination of factors. We do not use A9000R but rather FS9500 so I will try and give you some tools to check what AIX is doing.

    We have recently troubleshooted a performance issue and found poor response times related to CPU overload.

    I would check your CPU performance anyway. (lparstat 1 5 and vmstat 1 5 should be able to tell you at a glance)

    Then I always find iostat -Dl 1 5 useful to check how the disk is performing. (I normally look where disk avg_serv and q_full are high)

    This also helped us determine that some multipathing settings need to be tuned since the disk was performing well but AIX was throttling.

    • Have you encountered similar latency or I/O failure patterns after firmware upgrades? Not anything I can attribute to Firmware upgrade but rather workload patterns changing.
    • Are there recommended diagnostic utilities beyond the standard flashsystemcli and AIX errpt? You could use nmon in addition to the ones I mentioned in above. 
    • Could misaligned multipath settings or path failures be a contributing factor despite unchanged configurations? Yes, especially if the num_cmd_elems and/or queue_depth is incorrect.


    ------------------------------
    Regards

    Anwar Williams
    ------------------------------



Related Content