Global Storage

Global Storage Forum

Connect, collaborate, and stay informed with insights from across Storage

 View Only

  • 1.  Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Mon March 27, 2023 09:24 AM

    Hello all,

    We have an older Storwize V7000 which is used a little bit. One of the unused Mdisk went offline due to luck of spare. 

    New drives were added and new spares added but the mdisk remains offline. 

    From logs, when the last drive failed, the rebuild started on another drive which logged errors and the rebuild failed.

    5:mdisk5:offline:1:UserData_Pool:10.9TB:offline:raid6:0:128:tier_enterprise:no:no

    svcinfo lsarraymembergoals -delim : -bytes 5 2>&1

    5:mdisk5:0:60:1199706210304:tier_enterprise:10000:3:16::no:512:
    5:mdisk5:1:61:1199706210304:tier_enterprise:10000:3:9::no:512:
    5:mdisk5:2:62:1199706210304:tier_enterprise:10000:3:1::no:512:
    5:mdisk5:3:116:1199706210304:tier_enterprise:10000:3:7::no:512:
    5:mdisk5:4:64:1199706210304:tier_enterprise:10000:3:6::no:512:
    5:mdisk5:5:20:1199706210304:tier_enterprise:10000:1:20::no:512:
    5:mdisk5:6:66:1199706210304:tier_enterprise:10000:3:22::no:512:
    5:mdisk5:7:22:1199706210304:tier_enterprise:10000:1:10::no:512:
    5:mdisk5:8:21:1199706210304:tier_enterprise:10000:1:21::no:512:
    5:mdisk5:9:44:1199706210304:tier_enterprise:10000:2:22::no:512:
    5:mdisk5:10:70:1199706210304:tier_enterprise:10000::::no:512:  >>> Ghost ... 
    5:mdisk5:11:71:1199706210304:tier_enterprise:10000:3:23::no:512:

    We have assigned the spares manually and still have good Candidate drives in the machine. However, the rebuild didn't start automatically.

    The ghost drive is still a member and we we unable to change its use to Failed > Candidate >Unused 

    We also have tried to assign a new member to replace the ghost drive which is still member ID 10. The commands that we tried are as follow:

    1/ Assign a new drive and restart rebuilding on mdisk 5:

                  svctask charraymember -member 10 -newdrive 45 -immediate mdisk5 

    2/ Remove the ghost drive

      • svctask chdrive -use failed 70
      • svctask chdrive -use candidate 70
      • svctask chdrive -use unused 70

    Note that Drive 45 is a spare but it was a candidate before and nothing happened ... We have have other spares and candidate drives in the machine.

    Every time, we got an error message "cmmvc6539e the command cannot be initiated because the array does not have sufficient redundancy." 

    Can any one help on how to resolve the issue ?

    Thank you



    ------------------------------
    Justine Uwase
    ------------------------------


  • 2.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Tue March 28, 2023 12:31 PM

    Hola Justin!

    I haven't got into that situation before. Isn't the Web GUI telling you what to do? It should work if the manual method is missing a small step.

    I might try to recreate anyway... so just need 2 drives failed and try to fix the issue by using charraymember to assign candidate drives, right?



    ------------------------------
    Luis Lopez
    ------------------------------

    Original Message


  • 3.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Wed March 29, 2023 03:31 AM
    Hello,

    I just got involved with this case. Checking in the logs, I found 2 other failed rebuilds that never finished and on drives that belong to other mdisks. 

    I am not sure what happened to get these rebuild ongoing but I don't see this part in the logs ... 

    svcinfo lsarrayinitprogress -delim : 5 2>&1
    mdisk_id:mdisk_name:progress:estimated_completion_time
    5:mdisk5:100:

    svcinfo lsarraysyncprogress -delim : 5 2>&1
    mdisk_id:mdisk_name:progress:estimated_completion_time
    5:mdisk5:32:

    svcinfo lsarraymemberprogress -delim : 5 2>&1
    mdisk_id:mdisk_name:member_id:drive_id:task:new_drive_id:progress:estimated_completion_time
    5:mdisk5:7:22:rebuild::97:230324121000
    5:mdisk5:9:44:rebuild::32:230324180416

    Drive ID 97 is a member in mdisk8 and Drive ID 32 is in mdisk 3.

    Drive 22 and 44 seem to be online and OK.


    I am not sure how to delete these failed/suspended rebuilds.

    I am going to try to fail drive 32 and 97 .. hopefully, this will delete these failed rebuilds.

    Any other suggestions ?

    Thank you





    Original Message


  • 4.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Wed March 29, 2023 12:13 PM

    Thank you for the feedback .. please see more details on the issue below...



    ------------------------------
    Justine Uwase
    ------------------------------

    Original Message


  • 5.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Fri July 28, 2023 05:08 AM

    Hola Lopez,

    I think the final customer is doing something wrong but I have this issue on multiple machine now.

    The issue is that since the array is offline and the ghost drive loses its enclosure/slot infos and the attached error is fixed, nothing works anymore.

    All command to the array and drives fails ...

    The situation is getting out of hand .. 

    Please help



    ------------------------------
    Justine Uwase
    ------------------------------

    Original Message


  • 6.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Wed March 29, 2023 03:26 AM

    Hello Justine,

    please try:

    svctask chdrive -use unused -allowdegraded 70

    -allowdegraded(Optional)
    Permits permission for a change of drive use to continue, even if a hotspare drive is not available for the array that the drive is a member of. You cannot specify -allowdegraded and -task together. You cannot specify -allowdegraded when an array expansion is in progress.

    Use with care

    Greetings & Luck



    ------------------------------
    Patrik Groß
    ------------------------------

    Original Message


  • 7.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Wed March 29, 2023 12:15 PM

    Hello Patrik,

    Thank you for the feedback .. I will definitely try that ... Have you seen my update on the issue above ?

    Thank you



    ------------------------------
    Justine Uwase
    ------------------------------

    Original Message


  • 8.  RE: Storwize V7000 With a Ghost Drive and mdisk offline

    Posted Fri July 28, 2023 05:05 AM

    Hello,

    Sorry I took too long to reply. 

    The command didn't work and I think because the array is offline still .. we got the following error :" CMMVC6539E The command cannot be initiated because the array does not have sufficient redundancy."

    My question is how to we update the array info after replacing the failed drives ? 

    The array still thinks the drives are missing as its "lsarraymember" do get updated when the array is offline.

    On SVC or any attached storage, there is "detectmdisk" command, does any command exists to refresh the array configuration information ?

    Thank you



    ------------------------------
    Justine Uwase
    ------------------------------

    Original Message


Related Content