Primary Storage and IBM Spectrum Virtualize

Expand all | Collapse all

DS3524 Controller B always LU

  • 1.  DS3524 Controller B always LU

    Posted Tue December 10, 2019 02:45 PM
    I have a DS3524 where the second controller will not come back to normal. I've no idea what happened, I think controllers got moved around but what I do know is that controller A is working fine but controller B will not pick up and IP and constantly shows LU and a blue LED.

    How can I get this controller back online since I can't reach it?

    ------------------------------
    mike lewis
    ------------------------------


  • 2.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:03 PM
    Edited by Jochen Meixner Fri June 19, 2020 11:36 AM
    Hi Mike,
    mhhhh.... I have not used these machines for many years.
    Wasn't there an option to reset/reboot the other controller when you have access to the other one?

    I have found the following. It's not exactly the same, but perhaps it helps to find the the solution...
    https://www.ibm.com/support/pages/how-reset-ds3524-storage-controller-ibm-smart-analytics-systems

    Best regards,
    Jochen

    ------------------------------
    Jochen Meixner
    Principal Consultant Storage & Backup Solutions
    CONET Services GmbH
    Duesseldorf, Germany
    ------------------------------



  • 3.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:10 PM
    Found one more link, sounds very similar to your problem.
    https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014558015

    Be careful, the suggested solution erases the configuration of the controller...
    In one case the controller had to be replaced.

    ------------------------------
    Jochen Meixner
    Principal Consultant Storage & Backup Solutions
    CONET Services GmbH
    Duesseldorf, Germany
    ------------------------------



  • 4.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:13 PM
    Edited by mike lewis Fri June 19, 2020 11:36 AM
    Thanks for the input. I hoped this could work but it didn't. The error being that controller A cannot reach controller B. No issues on losing data, I've wiped the config and just trying to get the second controller back online.

    ------------------------------
    mike lewis
    ------------------------------



  • 5.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:20 PM
    Edited by mike lewis Fri June 19, 2020 11:37 AM

    I should also point out that when looking at DS manager, the second controller GUI is dark.

    BTW, the second link you shared is something I had found and tried. It's how I recovered the first controller but cannot get the second one to come online.

    ------------------------------
    mike lewis
    ------------------------------



  • 6.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:39 PM
    Hi Mike,
    ok. I assume you have set the system powerless and restarted it.
    Found some tips to update the firmware - difficult when the controller is unresponsive.

    One tip was to try to connect via the out-of-band connection which is the second port on the controller.
    Did you try this?

    ------------------------------
    Jochen Meixner
    Principal Consultant Storage & Backup Solutions
    CONET Services GmbH
    Duesseldorf, Germany
    ------------------------------



  • 7.  RE: DS3524 Controller B always LU

    Posted Tue December 10, 2019 07:43 PM
    I have not tried connecting directly but that's something I'll try.
    The firmware is fully up to date and yes, I've fully powered down, unplugged, you name it :).

    I appreciate the input.

    ------------------------------
    mike lewis
    ------------------------------



  • 8.  RE: DS3524 Controller B always LU

    Posted Wed December 11, 2019 09:52 AM
    Hi.
    The blue LED means that the device is ready to be removed from the enclosure
    You could try the following by the CLI interface on controller A
    1. execute the cmgrSetAltToFailed command

    2. Remove controller B wait 5 minutes and then reinsert it into the chassis.
    3. run the cmgrSetAltToOptimal command
    4. If the ip of controller B responds, then run on controller B:
         loadDebug
         lemClearLockdown


    ------------------------------
    Enrique Espinel
    ------------------------------



  • 9.  RE: DS3524 Controller B always LU

    Posted Wed December 11, 2019 02:59 PM
    I was able to run the commands but still don't see the controller in the DS manager and it has not picked up an IP.

    -> cmgrSetAltToFailed
    12/11/19-16:20:03 (tShellRem1): NOTE: CCM: takeover() setting MOS
    12/11/19-16:20:03 (tShellRem1): WARN: CCM: takeover() caught IconSendInfeasibleException Error , fai led to discard backup volumes on alt
    12/11/19-16:20:03 (tShellRem1): NOTE: buc controllerAltStateChanged
    12/11/19-16:20:03 (tShellRem1): WARN: Alt Ctl Reboot:
    Reboot CompID: 0x407
    Reboot reason: 0x6
    Reboot reason extra: 0x1
    12/11/19-16:20:03 (tShellRem1): NOTE: holding alt ctl in reset
    12/11/19-16:20:03 (tShellRem1): NOTE: Alt firmware running deasserted: 0 ms
    12/11/19-16:20:03 (tShellRem1): NOTE: HealthCheck: Alt Ctl: 1 Reset_Failure, state: 0 Start
    value = 0 = 0x0

    Unplugged.
    WAITED 30 MINUTES OR MORE
    Plugged Controller B back in.

    -> cmgrSetAltToOptimal
    12/11/19-19:54:39 (tShellRem1): NOTE: buc controllerAltStateChanged
    12/11/19-19:54:39 (tShellRem1): NOTE: releasing alt ctl from reset
    value = 0 = 0x0

    It just keeps fighting me. I also tried a replacement controller and it acts the same way.

    ------------------------------
    mike lewis
    ------------------------------



  • 10.  RE: DS3524 Controller B always LU

    Posted Sun December 15, 2019 04:46 PM
    Edited by mike lewis Fri June 19, 2020 11:37 AM
    Someone had contacted me offering to log into the device. I seem to have lost that email and I'm not sure I could get permission to allow it but I'd like to know more about this. Is this someone just offering to help or something that IBM is offering at a cost?

    Never did solve the problem. I cannot add or remove the controller, the chassis always thinks it's missing so cannot change the status in any way what so ever.

    ------------------------------
    mike lewis
    ------------------------------



  • 11.  RE: DS3524 Controller B always LU

    Posted Mon December 16, 2019 09:18 AM
    Hello, I offered to perform a remote verification to see more about the problem.
    This is free of charge, it is simply a help, for 18 years I was an IBM technical specialist in Ecuador.
    my mail: eespinel@gmail.com


    ------------------------------
    Enrique Espinel
    ------------------------------



  • 12.  RE: DS3524 Controller B always LU

    Posted Sun June 14, 2020 12:15 PM

    Hi again,

    Thank you for your help in the past. It took me a long time to find this old thread. The problem came back. I recently cleared the config from the GUI and since then, cannot access the controllers anymore.

    The system was working fine until I cleared everything from the GUI to get it ready to re-configure.
    Now I've lost access and still cannot get into it other than telnet.

    Using telnet, I've also tried other things;

    sysWipe
    sysWipeAllConfigData
    lemClearLockdownbut

    Nothing has it seen in the DS manager anymore.

    Strange that this happens again.



    ------------------------------
    mike lewis
    ------------------------------