Global Storage

Global Storage Forum

Connect, collaborate, and stay informed with insights from across Storage

 View Only

  • 1.  Trying to revive a DS6800

    Posted Mon July 21, 2025 09:42 PM

    This DS6800 was resurrected 5 years ago and worked until it was powered down 2 years ago. Now it will not initialize. Batteries are good.  Both Kona cards fail to open DA ports.  Here's a copy of the IML log from 5 years ago when it worked. Copy of the failure follows.

    5f637d09  Arrowhead reset complete.

    5f637d09

    Arrowhead VPD Firmware Level = 201002100011

    5f637d09  Firmware File Level = 201002100011

    5f637d09

    Arrowhead download NOT required.

    5f637d09  DA_ADAPINIT ISS_CPSS_SETUP done

    5f637d09  daiConfigOpenAdap done, rc = 0

    5f637d09  lnxInitArrowhead, WAIT For Enable Ports to complete

    5f637d09  Recovery: no assigned slots

    5f637d09  dasdsHarvestVPDCmplt: VPD data for Adapter 1001

    5f637d09  dasdsHarvestVPDCmplt: Part Number = 0kon4.31

    5f637d09  dasdsHarvestVPDCmplt: EC Level = NoHursley0

    5f637d09  dasdsHarvestVPDCmplt: Card Serial Number = Skon4.31YM10MY3B1880

    5f637d09  dasdsHarvestVPDCmplt: World Wide Name  = 50050763100037DB

    5f637d09  dasdsHarvestVPDCmplt: ROS level = 201002100011

    5f637d16  lnxInitArrowhead, Enable Ports Complete, go ahead

    Here's the failure.

    556e0d5e  Arrowhead reset complete.
    556e0d5e

    Arrowhead VPD Firmware Level = 201002100011
    556e0d5e  Firmware File Level = 201002100011
    556e0d5e

    Arrowhead download NOT required.


    556e0d5e  DA_ADAPINIT ISS_CPSS_SETUP done
    556e0d5e  daiConfigOpenAdap done, rc = 0
    556e0d5e  lnxInitArrowhead, WAIT For Enable Ports to complete
    556e0d62  AR sending ER_IOA_AR_RESET_INIT_FAILED type specific EIR for adapter 0x1001
    556e0d62  Following registers are little Endian
    556e0d62  DoorBell In Register = 0x00000000, DoorBell Out Register = 0x88000000
    556e0d62  Control Register = 0x00000000,  Status Register = 0xc0000000
    556e0d62  Firmware Error Register = 0x20000000,  Diagnostic Offset Register = 0x00086000
    556e0d62  Diagnostic offset data = 0x00000000 0x00000000 0x00000000
    556e0d62  *** ERP *** MDE CALL from dasdsutil.c dasdsCallERD line 1749
    556e0d62  *** ERP *** Error 1 - Processing Sync IOA Error:IOA Error    suberrortype:Hardware Error    suberr:ER_IOA_ARROWHEAD    tspec:ER_IOA_AR_RESET_INIT_FAILED    adID:1001
    556e0d62  *** ER: Saving EIR #0 to archive slot #0 (out of 30) [87128].
    556e0d62  *** The DA reason code is d2 ***
    556e0d62  *** ER: translating Bering/Arrowhead recovery (1001) ***
    556e0d62  *** ER: Rebooting server 0 ***
    556e0d62  *** ERP *** Error 1 - errRA Recovery Action: ER_SINGLE_REBOOT
    556e0d62  *******************************
    556e0d62  *****                    ******
    556e0d62  ***** Rebooting This CEC ******
    556e0d62  *****                    ******
    556e0d62  *******************************



    ------------------------------
    Dave Gallenberger
    ------------------------------


  • 2.  RE: Trying to revive a DS6800

    Posted Fri August 29, 2025 12:22 PM

    The kona card should be bad from my experience too.



    ------------------------------
    MICHIO IKEDA
    ------------------------------

    Original Message


  • 3.  RE: Trying to revive a DS6800

    Posted Tue September 02, 2025 09:45 AM

    Hello @MICHIO IKEDA

    The failure you're experiencing with the DS6800, as described in the POSTED log, indicates that the Arrowhead reset attempt was unsuccessful, resulting in an error (ER_IOA_ARROWHEAD). This error suggests a problem with the IO Cage (I/O Cage) or the Arrowhead component within the DS6800. Artifacts in the log suggest that the DCACML (DCAC Loopback Cage Adapter) and its associated ports might be the root cause. The error registers (DoorBell In, DoorBell Out, Control, Status, and Firmware Error Registers) don't seem to be in the expected state, with unexpected values. In particular, the DCACML Register 0x20000000 indicating a Hardware Error highlights an issue that's not resolved after attempting reset. This issue appears to be unrelated to the initialization status from 5 years ago and is related to active IOCages. As you've already confirmed that the Kona card batteries are good, and the problem persists even after proper initialization, it might be a physical issue with the DCACML. For a more in-depth analysis, IBM Support Engineers at the IBM Technical Support Center (TSC) can examine detailed logged data (EIRs) and communicate with the system remotely to identify the precise cause of the error and recommend further troubleshooting steps. To get in touch with IBM Support, initiate a hardware support request through the IBM Website for Technical Support. Details can be found at: IBM Hardware Servicing Use this resource to initiate a support case, and provide them with detailed log information. This process allows support professionals to analyze your issue comprehensively and guide you through potential solutions. Resources: - "IBM Hardware Servicing" page - IBM Website for Technical Support: IBM Contact Information

    Other cause: could be a System forced to reboot, final confirmation of a nonrecoverable fault

    You may also want to run your IML Log on another AI system
    More info can be found in: #IBM Redbooks
     IBM Redbooks - International Technical Support Organization
     IBM System Storage DS6000 Series: 
     Architecture and Implementation
    IBM System Storage DS6000 Series: Architecture and Implementation
                                        


    ------------------------------
    Thomas Mertens
    Medford, Wi U.S.A (Summer)
    Florida (Winter) U.S.A.
    ------------------------------

    Original Message


  • 4.  RE: Trying to revive a DS6800

    Posted Wed September 03, 2025 08:42 PM

    Hi, that's a great HW analysis Tom. 
    I don't want to be the one who bring bad news, but I won't think IBM Support would help in this case. The IBM DS6S800 is EOS from 2015. 
    I believe they can try a "Best Effort" mode, but that won't guarenteed a recovery. I am telling this because of a similar situation, recently, with an IBM DS6800 and IBM was unable to bring it back online. We managed to use an "old" IBM DS8000 and use it to connect to the zSeries.
    But in conclusion, due the condition of EOS for an IBM DS6800, I don't think will be possible to obtain proper support: https://www.globaltpm.com/ibm-ds-storage-eosl-dates/
    Best of luck Dave.



    ------------------------------
    Angel Alejandro Rivas Pazos
    Infrastructure Technical Leader | Storage
    GBM
    Ciudad de Panama
    (+507) 63916149
    ------------------------------

    Original Message


  • 5.  RE: Trying to revive a DS6800

    Posted Wed September 03, 2025 12:29 PM

    Hello @MICHIO IKEDA

    After some research with AI assistant, here i gave me these results:

    The failure you're experiencing with the DS6800, as described in the POSTED log, indicates that the Arrowhead reset attempt was unsuccessful, resulting in an error (ER_IOA_ARROWHEAD). This error suggests a problem with the IO Cage (I/O Cage) or the Arrowhead component within the DS6800. Artifacts in the log suggest that the DCACML (DCAC Loopback Cage Adapter) and its associated ports might be the root cause. The error registers (DoorBell In, DoorBell Out, Control, Status, and Firmware Error Registers) don't seem to be in the expected state, with unexpected values. In particular, the DCACML Register 0x20000000 indicating a Hardware Error highlights an issue that's not resolved after attempting reset. This issue appears to be unrelated to the initialization status from 5 years ago and is related to active IOCages. As you've already confirmed that the Kona card batteries are good, and the problem persists even after proper initialization, it might be a physical issue with the DCACML. For a more in-depth analysis, IBM Support Engineers at the IBM Technical Support Center (TSC) can examine detailed logged data (EIRs) and communicate with the system remotely to identify the precise cause of the error and recommend further troubleshooting steps. To get in touch with IBM Support, initiate a hardware support request through the IBM Website for Technical Support. Details can be found at: IBM Hardware Servicing Use this resource to initiate a support case, and provide them with detailed log information. This process allows support professionals to analyze your issue comprehensively and guide you through potential solutions. Resources: - "IBM Hardware Servicing" page - IBM Website for Technical Support: IBM Contact Information

    Also info may be found in: IBM System Storage DS6000 Series: Architecture and Implementation

    Let me know if this helps ( I am not a Storage Specialist)

    Best Regards

    Thomas Mertens



    ------------------------------
    Thomas Mertens
    Medford, Wi U.S.A (Summer)
    Florida (Winter) U.S.A.
    ------------------------------


  • 6.  RE: Trying to revive a DS6800

    Posted Wed November 12, 2025 01:04 AM

    @Thomas Mertens Sorry for late reply. I apologize for not noticing your post sooner and for the delayed response. Yes, I believe your analysis is correct. This error should be caused by a failure in the arrowhead module inside the Kona card (controller card). To resolve the failure, the Kona card needs to be replaced. Based on my experience several years ago, there were many cases of NDP (New Part Defective) with these components, and even after replacement, this error occurred again with several units. Replacing it with a non-defective part should resolve the failure.

     
    Translated with DeepL.com (free version)


    ------------------------------
    MICHIO IKEDA
    ------------------------------

    Original Message


  • 7.  RE: Trying to revive a DS6800

    Posted Wed November 12, 2025 08:37 AM

    I agree it really looks like a hardware problem. It bothers me that it happening on both controllers the same way after sitting a couple years. I talked to another x-IBM guy and he thought the D2 return code meant nvram was corrupted. He did not have the procedure to clear it.



    ------------------------------
    Dave Gallenberger
    ------------------------------

    Original Message