ECE reports a pdisk as missing/replace. So customer needs to replace this broken pdisk. But "mmvdisk pdisk replace --prepare --recovery-group RGNAME --pdisk PDISKNAME" command failed with error "Location of pdisk PDISKNAME of recovery group RGNAME is not known". And because this disk is missing, so the light can't be turned on and customer doesn't know which physical disk to replace.
Each ECE node in this cluster has one SCSI HBA card two links. One link to go the front side of the server and is connected to 24 hard disks. Another link is connected to the rear side of the server and connected to 13 pdisks. Checked the slotmap.yaml being used:
[root@ECE-A01 ~]# cd /usr/lpp/mmfs/data/gems
[root@ECE-A01 gems]# ls
README.gems slotmap.yaml
[root@ECE-A01 gems]# cat slotmap.yaml
version: 2
controllers:
- controller: 0
eids:
- eidx: 0
slots:
- {storcli-slot: 0, machine-slot: 0}
- {storcli-slot: 1, machine-slot: 1}
- {storcli-slot: 2, machine-slot: 2}
- {storcli-slot: 3, machine-slot: 3}
- {storcli-slot: 4, machine-slot: 4}
- {storcli-slot: 5, machine-slot: 5}
- {storcli-slot: 6, machine-slot: 6}
- {storcli-slot: 7, machine-slot: 7}
- {storcli-slot: 8, machine-slot: 8}
- {storcli-slot: 9, machine-slot: 9}
- {storcli-slot: 10, machine-slot: 10}
- {storcli-slot: 11, machine-slot: 11}
- {storcli-slot: 12, machine-slot: 12}
- {storcli-slot: 13, machine-slot: 13}
- {storcli-slot: 14, machine-slot: 14}
- {storcli-slot: 15, machine-slot: 15}
- {storcli-slot: 16, machine-slot: 16}
- {storcli-slot: 17, machine-slot: 17}
- {storcli-slot: 18, machine-slot: 18}
- {storcli-slot: 19, machine-slot: 19}
- {storcli-slot: 20, machine-slot: 20}
- {storcli-slot: 21, machine-slot: 21}
- {storcli-slot: 22, machine-slot: 22}
- {storcli-slot: 23, machine-slot: 23}
- {storcli-slot: 24, machine-slot: 24}
- {storcli-slot: 25, machine-slot: 25}
- {storcli-slot: 26, machine-slot: 26}
- {storcli-slot: 27, machine-slot: 27}
- {storcli-slot: 28, machine-slot: 28}
- {storcli-slot: 29, machine-slot: 29}
- {storcli-slot: 30, machine-slot: 30}
- {storcli-slot: 31, machine-slot: 31}
- {storcli-slot: 32, machine-slot: 32}
- {storcli-slot: 33, machine-slot: 33}
- {storcli-slot: 34, machine-slot: 34}
- {storcli-slot: 35, machine-slot: 35}
- {storcli-slot: 37, machine-slot: 37}
- {storcli-slot: 38, machine-slot: 38}
- {storcli-slot: 39, machine-slot: 39}
The above config means there is only 1 link and it's connected to 39 hard disks(including system OS hard disk). This is wrong.
Run storcli64 command to check the hardware topology:
[root@ECE-A01 gems]# /opt/MegaRAID/storcli/storcli64 /call show|more
Generating detailed summary of the adapter, it may take a while to complete.
...
JBOD Drives = 37
JBOD LIST :
=========
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------
8:0 13 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:1 28 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:2 20 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:3 25 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:4 18 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:5 21 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:6 14 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:7 29 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:8 16 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:9 34 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:10 19 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:11 15 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:12 17 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:13 22 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:14 42 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:15 27 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:16 23 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:17 24 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:18 26 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:19 31 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:20 36 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:21 40 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:22 30 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
8:23 43 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:0 37 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:1 35 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:2 45 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:3 39 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:4 47 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:5 32 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:6 33 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:7 38 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:8 48 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:9 44 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:10 41 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:11 46 JBOD - 14.552 TB SATA HDD N N 512B ST16000NM001G-2KK103 U -
9:17 10 JBOD - 447.130 GB SATA SSD N N 512B INTEL SSDSC2KB480G8 U -
----------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|Onln=Online|
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
So there are 2 links(8,9) with 24 and 13 pdisks attached respectively. We need to generate a correct mapping file using ecedrivemapping.
[root@ECE-A01 gems]# ecedrivemapping
ERROR:root:This tool requires the IBM Spectrum Scale daemon to be stopped on this node to run correctly. Please run the mmshutdown command on this node and try again.
[root@ECE-A01 gems]# mmshutdown -a
When ecedrivemapping is running, you need to check the lights on the server.. ecedrivemapping will turn on the light for each slot one by one. When a light for a pdisk is turned on, you need to input the label of the pdisk(it could only accept number):
[root@ECE-A01 gems]# ecedrivemapping --mode lmr
>>> 39 Lmr drives were detected
>>> Enter the first slot that can be used for Lmr drives: 1
>>> Enter the last slot that can be used for Lmr drives: 37
Now blinking path /c0/e8/s0 <---- the light on this pdisk will be blinking
>>> Enter the slot number: 1 <--- each pdisk should have a label. Input the number of this label.
Now blinking path /c0/e8/s1
>>> Enter the slot number: 2
Now blinking path /c0/e8/s2
>>> Enter the slot number: 3
Now blinking path /c0/e8/s3
>>> Enter the slot number: 4
...
>>> Enter the slot number: 28
Now blinking path /c0/e9/s4
>>> Enter the slot number: 29
Now blinking path /c0/e9/s5
>>> Enter the slot number: 30
Now blinking path /c0/e9/s6
>>> Enter the slot number: 31
Now blinking path /c0/e9/s7
>>> Enter the slot number: 32
...
Now blinking path /c0/e9/s17
>>> Enter the slot number: 37
Backing up /usr/lpp/mmfs/data/gems/slotmap.yaml to /usr/lpp/mmfs/data/gems/slotmap.yaml_backup1
Wrote to /usr/lpp/mmfs/data/gems/slotmap.yaml
[root@ECE-A01 gems]#
The pdisks are located on both front and rear side of the server... To make it easy to understand, we will change the "machine-slot" to something like "F1", "B3" which means 1st slot on front side and 3rd slot on rear side... "machine-slot" is used for admin to identify the pdisk location and the actual underlying scsi command will use "storcli-slot" instead.
So we change slotmap.yaml from:
version: 2
controllers:
- controller: 0
eids:
- eidx: 0
slots:
- {storcli-slot: 0, machine-slot: 1}
- {storcli-slot: 1, machine-slot: 2}
- {storcli-slot: 2, machine-slot: 3}
- {storcli-slot: 3, machine-slot: 4}
- {storcli-slot: 4, machine-slot: 5}
- {storcli-slot: 5, machine-slot: 6}
- {storcli-slot: 6, machine-slot: 7}
- {storcli-slot: 7, machine-slot: 8}
- {storcli-slot: 8, machine-slot: 9}
- {storcli-slot: 9, machine-slot: 10}
- {storcli-slot: 10, machine-slot: 11}
- {storcli-slot: 11, machine-slot: 12}
- {storcli-slot: 12, machine-slot: 13}
- {storcli-slot: 13, machine-slot: 14}
- {storcli-slot: 14, machine-slot: 15}
- {storcli-slot: 15, machine-slot: 16}
- {storcli-slot: 16, machine-slot: 17}
- {storcli-slot: 17, machine-slot: 18}
- {storcli-slot: 18, machine-slot: 19}
- {storcli-slot: 19, machine-slot: 20}
- {storcli-slot: 20, machine-slot: 21}
- {storcli-slot: 21, machine-slot: 22}
- {storcli-slot: 22, machine-slot: 23}
- {storcli-slot: 23, machine-slot: 24}
- eidx: 1
slots:
- {storcli-slot: 0, machine-slot: 25}
- {storcli-slot: 1, machine-slot: 26}
- {storcli-slot: 2, machine-slot: 27}
- {storcli-slot: 3, machine-slot: 28}
- {storcli-slot: 4, machine-slot: 29}
- {storcli-slot: 5, machine-slot: 30}
- {storcli-slot: 6, machine-slot: 31}
- {storcli-slot: 7, machine-slot: 32}
- {storcli-slot: 8, machine-slot: 33}
- {storcli-slot: 9, machine-slot: 34}
- {storcli-slot: 10, machine-slot: 35}
- {storcli-slot: 11, machine-slot: 36}
- {storcli-slot: 17, machine-slot: 37}
to:
version: 2
controllers:
- controller: 0
eids:
- eidx: 0
slots:
- {storcli-slot: 0, machine-slot: F1}
- {storcli-slot: 1, machine-slot: F2}
- {storcli-slot: 2, machine-slot: F3}
- {storcli-slot: 3, machine-slot: F4}
- {storcli-slot: 4, machine-slot: F5}
- {storcli-slot: 5, machine-slot: F6}
- {storcli-slot: 6, machine-slot: F7}
- {storcli-slot: 7, machine-slot: F8}
- {storcli-slot: 8, machine-slot: F9}
- {storcli-slot: 9, machine-slot: F10}
- {storcli-slot: 10, machine-slot: F11}
- {storcli-slot: 11, machine-slot: F12}
- {storcli-slot: 12, machine-slot: F13}
- {storcli-slot: 13, machine-slot: F14}
- {storcli-slot: 14, machine-slot: F15}
- {storcli-slot: 15, machine-slot: F16}
- {storcli-slot: 16, machine-slot: F17}
- {storcli-slot: 17, machine-slot: F18}
- {storcli-slot: 18, machine-slot: F19}
- {storcli-slot: 19, machine-slot: F20}
- {storcli-slot: 20, machine-slot: F21}
- {storcli-slot: 21, machine-slot: F22}
- {storcli-slot: 22, machine-slot: F23}
- {storcli-slot: 23, machine-slot: F24}
- eidx: 1
slots:
- {storcli-slot: 0, machine-slot: B1}
- {storcli-slot: 1, machine-slot: B2}
- {storcli-slot: 2, machine-slot: B3}
- {storcli-slot: 3, machine-slot: B4}
- {storcli-slot: 4, machine-slot: B5}
- {storcli-slot: 5, machine-slot: B6}
- {storcli-slot: 6, machine-slot: B7}
- {storcli-slot: 7, machine-slot: B8}
- {storcli-slot: 8, machine-slot: B9}
- {storcli-slot: 9, machine-slot: B10}
- {storcli-slot: 10, machine-slot: B11}
- {storcli-slot: 11, machine-slot: B12}
- {storcli-slot: 17, machine-slot: B13}
After this mapping is generated, then copy it to all other ECE servers... This will avoid running ecedrivemapping on each node... This is because all ECE servers in the same RG should have the exact same hardware config and topology.
After slotmap.yaml are copied to each ECE node, you need to run "tslsenclslot" to verify the current disk location mapping on each node.
[root@ECE-A01 gems]# tslsenclslot -a
tslsenclslot:trace:HEADER:function:message:
tslsenclslot:log:HEADER:message:
tslsenclslot:slot:HEADER:LocationCode:EnclosureType:EnclosureID:SlotHandle:Capabilities:Occupied:Swapped:Power:Identify:Replace:Fail:FailureDomains:PortAddresses:LogicalUnits:
tslsenclslot:slot:0:210235A3THH219000545-B1:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-0::naa.5000C500DC0C22D5:
tslsenclslot:slot:0:210235A3THH219000545-B2:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-1::naa.5000C500DC0BF4BD:
tslsenclslot:slot:0:210235A3THH219000545-B3:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-2:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-2::naa.5000C500DC0F4B3E:
tslsenclslot:slot:0:210235A3THH219000545-B4:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-3:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-3::naa.5000C500DC0C06FB:
tslsenclslot:slot:0:210235A3THH219000545-B5:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-4:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-4::naa.5000C500DC0CB493:
tslsenclslot:slot:0:210235A3THH219000545-B6:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-5:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-5::naa.5000C500DC0F69D4:
tslsenclslot:slot:0:210235A3THH219000545-B7:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-6:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-6::naa.5000C500DC476D61:
tslsenclslot:slot:0:210235A3THH219000545-B8:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-7:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-7::naa.5000C500DC0BC11E:
tslsenclslot:slot:0:210235A3THH219000545-B9:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-8:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-8::naa.5000C500DBFDD2F5:
tslsenclslot:slot:0:210235A3THH219000545-B10:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-9:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-9::naa.5000C500DC47DF6D:
tslsenclslot:slot:0:210235A3THH219000545-B11:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-10:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-10::naa.5000C500DC0C196C:
tslsenclslot:slot:0:210235A3THH219000545-B12:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-11:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-11::naa.5000C500DC0C4307:
tslsenclslot:slot:0:210235A3THH219000545-B13:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-1-17:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-17::naa.55CD2E41547E7DDD:
tslsenclslot:slot:0:210235A3THH219000545-F1:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-0::naa.5000C500DC495765:
tslsenclslot:slot:0:210235A3THH219000545-F2:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-1::naa.5000C500DC47269C:
tslsenclslot:slot:0:210235A3THH219000545-F3:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-2:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-2::naa.5000C500DC0B82D1:
tslsenclslot:slot:0:210235A3THH219000545-F4:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-3:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-3::naa.5000C500DC0D8E22:
tslsenclslot:slot:0:210235A3THH219000545-F5:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-4:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-4::naa.5000C500DC0C4DDD:
tslsenclslot:slot:0:210235A3THH219000545-F6:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-5:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-5::naa.5000C500DC46F52C:
tslsenclslot:slot:0:210235A3THH219000545-F7:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-6:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-6::naa.5000C500DC0BEDC9:
tslsenclslot:slot:0:210235A3THH219000545-F8:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-7:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-7::naa.5000C500DC0BEFC3:
tslsenclslot:slot:0:210235A3THH219000545-F9:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-8:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-8::naa.5000C500DC0D89C3:
tslsenclslot:slot:0:210235A3THH219000545-F10:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-9:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-9::naa.5000C500DC0D4A04:
tslsenclslot:slot:0:210235A3THH219000545-F11:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-10:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-10::naa.5000C500DC490CDB:
tslsenclslot:slot:0:210235A3THH219000545-F12:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-11:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-11::naa.5000C500DC490F80:
tslsenclslot:slot:0:210235A3THH219000545-F13:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-12:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-12::naa.5000C500DC3BF6F1:
tslsenclslot:slot:0:210235A3THH219000545-F14:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-13:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-13::naa.5000C500DC4BD365:
tslsenclslot:slot:0:210235A3THH219000545-F15:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-14:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-14::naa.5000C500DC4BEC6A:
tslsenclslot:slot:0:210235A3THH219000545-F16:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-15:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-15::naa.5000C500DC483AF6:
tslsenclslot:slot:0:210235A3THH219000545-F17:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-16:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-16::naa.5000C500DC494B32:
tslsenclslot:slot:0:210235A3THH219000545-F18:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-17:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-17::naa.5000C500DC489061:
tslsenclslot:slot:0:210235A3THH219000545-F19:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-18:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-18::naa.5000C500DC4201A1:
tslsenclslot:slot:0:210235A3THH219000545-F20:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-19:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-19::naa.5000C500DC46F361:
tslsenclslot:slot:0:210235A3THH219000545-F21:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-20:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-20::naa.5000C500DC3DA39F:
tslsenclslot:slot:0:210235A3THH219000545-F22:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-21:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-21::naa.5000C500DC477CE9:
tslsenclslot:slot:0:210235A3THH219000545-F23:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-22:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-22::naa.5000C500DC48F373:
tslsenclslot:slot:0:210235A3THH219000545-F24:AVAGO_MegaRAID:SKB3567150:lmr!SKB3567150!SLOT!0-0-23:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-23::naa.5000C500DC460F48:
The above is correct now. You can also use "ecedrivemapping --report --mode lmr" to verify the slot mapping:
# ecedrivemapping --report --mode lmr
Displaying current storcli-slot to machine-slot map file:
------------------------------------------
Controller: 0
------------------------------------------
storcli-slot : 0 => machine-slot : F1
storcli-slot : 1 => machine-slot : F2
storcli-slot : 2 => machine-slot : F3
storcli-slot : 3 => machine-slot : F4
storcli-slot : 4 => machine-slot : F5
storcli-slot : 5 => machine-slot : F6
storcli-slot : 6 => machine-slot : F7
storcli-slot : 7 => machine-slot : F8
storcli-slot : 8 => machine-slot : F9
storcli-slot : 9 => machine-slot : F10
storcli-slot : 10 => machine-slot : F11
storcli-slot : 11 => machine-slot : F12
storcli-slot : 12 => machine-slot : F13
storcli-slot : 13 => machine-slot : F14
storcli-slot : 14 => machine-slot : F15
storcli-slot : 15 => machine-slot : F16
storcli-slot : 16 => machine-slot : F17
storcli-slot : 17 => machine-slot : F18
storcli-slot : 18 => machine-slot : F19
storcli-slot : 19 => machine-slot : F20
storcli-slot : 20 => machine-slot : F21
storcli-slot : 21 => machine-slot : F22
storcli-slot : 22 => machine-slot : F23
storcli-slot : 23 => machine-slot : F24
storcli-slot : 0 => machine-slot : B1
storcli-slot : 1 => machine-slot : B2
storcli-slot : 2 => machine-slot : B3
storcli-slot : 3 => machine-slot : B4
storcli-slot : 4 => machine-slot : B5
storcli-slot : 5 => machine-slot : B6
storcli-slot : 6 => machine-slot : B7
storcli-slot : 7 => machine-slot : B8
storcli-slot : 8 => machine-slot : B9
storcli-slot : 9 => machine-slot : B10
storcli-slot : 10 => machine-slot : B11
storcli-slot : 11 => machine-slot : B12
storcli-slot : 17 => machine-slot : B13
But the disk slot mapping for ECE-A03 is wrong:
tslsenclslot:slot:0:210235A3THH219000533-0-1-12:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-12:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-12::naa.50000C500DC483520:
tslsenclslot:slot:0:210235A3THH219000533-0-1-13:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-13:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-13::naa.50000C500DC0D583D:
...
tslsenclslot:slot:0:210235A3THH219000533-0-1-21:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-21:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-21::naa.50000C500DC4922B4:
tslsenclslot:slot:0:210235A3THH219000533-0-1-22:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-22:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-22::naa.50000C500DC47EF97:
tslsenclslot:slot:0:210235A3THH219000533-0-1-23:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-23:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-23::naa.50000C500DC46F6CA:
...
tslsenclslot:slot:0:210235A3THH219000533-B1:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-0::naa.5000C500DCC0CC6EA:
tslsenclslot:slot:0:210235A3THH219000533-B2:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-1::naa.5000C500DCC0C3A1F:
...
tslsenclslot:slot:0:210235A3THH219000533-B14:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-18:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-18::naa.5000C5000DC4A7B8D:
tslsenclslot:slot:0:210235A3THH219000533-B15:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-19:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-19::naa.5000C5000DC4A80F8:
tslsenclslot:slot:0:210235A3THH219000533-F1:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-0::naa.5000C500DCC467D83:
tslsenclslot:slot:0:210235A3THH219000533-F2:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-1::naa.5000C500DCC48FEA1:
...
tslsenclslot:slot:0:210235A3THH219000533-F22:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-21:0x01001100:0:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-21:::
tslsenclslot:slot:0:210235A3THH219000533-F23:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-22:0x01001100:0:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-22:::
This is because the server manufacturer connected the 1st link to the rear side, and 2nd link to front side.
So you have two choices here:
1. reconnect to switch the SAS links of this server
2. Modify the slotmap.yaml on this node.
We choose the 2nd. So slotmap.yaml is changed on this node from:
version: 2
controllers:
- controller: 0
eids:
- eidx: 0 ---> change to 1
slots:
- {storcli-slot: 0, machine-slot: F1}
- {storcli-slot: 1, machine-slot: F2}
- {storcli-slot: 2, machine-slot: F3}
- {storcli-slot: 3, machine-slot: F4}
- {storcli-slot: 4, machine-slot: F5}
- {storcli-slot: 5, machine-slot: F6}
- {storcli-slot: 6, machine-slot: F7}
- {storcli-slot: 7, machine-slot: F8}
- {storcli-slot: 8, machine-slot: F9}
- {storcli-slot: 9, machine-slot: F10}
- {storcli-slot: 10, machine-slot: F11}
- {storcli-slot: 11, machine-slot: F12}
- {storcli-slot: 12, machine-slot: F13}
- {storcli-slot: 13, machine-slot: F14}
- {storcli-slot: 14, machine-slot: F15}
- {storcli-slot: 15, machine-slot: F16}
- {storcli-slot: 16, machine-slot: F17}
- {storcli-slot: 17, machine-slot: F18}
- {storcli-slot: 18, machine-slot: F19}
- {storcli-slot: 19, machine-slot: F20}
- {storcli-slot: 20, machine-slot: F21}
- {storcli-slot: 21, machine-slot: F22}
- {storcli-slot: 22, machine-slot: F23}
- {storcli-slot: 23, machine-slot: F24}
- eidx: 1 --> change to 0
slots:
- {storcli-slot: 0, machine-slot: B1}
- {storcli-slot: 1, machine-slot: B2}
- {storcli-slot: 2, machine-slot: B3}
- {storcli-slot: 3, machine-slot: B4}
- {storcli-slot: 4, machine-slot: B5}
- {storcli-slot: 5, machine-slot: B6}
- {storcli-slot: 6, machine-slot: B7}
- {storcli-slot: 7, machine-slot: B8}
- {storcli-slot: 8, machine-slot: B9}
- {storcli-slot: 9, machine-slot: B10}
- {storcli-slot: 10, machine-slot: B11}
- {storcli-slot: 11, machine-slot: B12}
- {storcli-slot: 17, machine-slot: B13}
Now run "tslsenclslot" again on this node, and the mapping is correct now:
[root@ECE-A03 gems]# tslsenclslot -a
tslsenclslot:trace:HEADER:function:message:
tslsenclslot:log:HEADER:message:
tslsenclslot:slot:HEADER:LocationCode:EnclosureType:EnclosureID:SlotHandle:Capabilities:Occupied:Swapped:Power:Identify:Replace:Fail:FailureDomains:PortAddresses:LogicalUnits:
tslsenclslot:slot:0:210235A3THH219000533-B1:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-0::naa.5000C500DC467D83:
tslsenclslot:slot:0:210235A3THH219000533-B2:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-1::naa.5000C500DC48FEA1:
tslsenclslot:slot:0:210235A3THH219000533-B3:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-2:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-2::naa.5000C500DC3CA81C:
tslsenclslot:slot:0:210235A3THH219000533-B4:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-3:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-3::naa.5000C500DC0C1639:
tslsenclslot:slot:0:210235A3THH219000533-B5:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-4:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-4::naa.5000C500DC4A5F49:
tslsenclslot:slot:0:210235A3THH219000533-B6:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-5:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-5::naa.5000C500DC484CA6:
tslsenclslot:slot:0:210235A3THH219000533-B7:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-6:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-6::naa.5000C500DC48A179:
tslsenclslot:slot:0:210235A3THH219000533-B8:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-7:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-7::naa.5000C500DC0BE9E4:
tslsenclslot:slot:0:210235A3THH219000533-B9:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-8:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-8::naa.5000C500DC38F85D:
tslsenclslot:slot:0:210235A3THH219000533-B10:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-9:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-9::naa.5000C500DC0CE5E0:
tslsenclslot:slot:0:210235A3THH219000533-B11:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-10:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-10::naa.5000C500DC1BC9B0:
tslsenclslot:slot:0:210235A3THH219000533-B12:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-11:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-11::naa.5000C500DC478708:
tslsenclslot:slot:0:210235A3THH219000533-B13:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-17:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-17::naa.55CD2E41547E6642:
tslsenclslot:slot:0:210235A3THH219000533-B14:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-18:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-18::naa.55CD2E41547E6F46:
tslsenclslot:slot:0:210235A3THH219000533-B15:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-0-19:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-0-19::naa.55CD2E41547E6F34:
tslsenclslot:slot:0:210235A3THH219000533-F1:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-0:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-0::naa.5000C500DC0CC6EA:
tslsenclslot:slot:0:210235A3THH219000533-F2:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-1:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-1::naa.5000C500DC0C3A1F:
tslsenclslot:slot:0:210235A3THH219000533-F3:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-2:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-2::naa.5000C500DC47731F:
tslsenclslot:slot:0:210235A3THH219000533-F4:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-3:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-3::naa.5000C500DC469EDD:
tslsenclslot:slot:0:210235A3THH219000533-F5:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-4:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-4::naa.5000C500DC465668:
tslsenclslot:slot:0:210235A3THH219000533-F6:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-5:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-5::naa.5000C500DC475F59:
tslsenclslot:slot:0:210235A3THH219000533-F7:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-6:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-6::naa.5000C500DC1BA26E:
tslsenclslot:slot:0:210235A3THH219000533-F8:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-7:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-7::naa.5000C500DC4A4771:
tslsenclslot:slot:0:210235A3THH219000533-F9:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-8:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-8::naa.5000C500DC0C5F39:
tslsenclslot:slot:0:210235A3THH219000533-F10:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-9:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-9::naa.5000C500DC0C1762:
tslsenclslot:slot:0:210235A3THH219000533-F11:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-10:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-10::naa.5000C500DC0C14BE:
tslsenclslot:slot:0:210235A3THH219000533-F12:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-11:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-11::naa.5000C500DC0BF653:
tslsenclslot:slot:0:210235A3THH219000533-F13:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-12:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-12::naa.5000C500DC483520:
tslsenclslot:slot:0:210235A3THH219000533-F14:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-13:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-13::naa.5000C500DC0D583D:
tslsenclslot:slot:0:210235A3THH219000533-F15:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-14:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-14::naa.5000C500DC490203:
tslsenclslot:slot:0:210235A3THH219000533-F16:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-15:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-15::naa.5000C500DC467843:
tslsenclslot:slot:0:210235A3THH219000533-F17:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-16:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-16::naa.5000C500DC0C4615:
tslsenclslot:slot:0:210235A3THH219000533-F18:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-17:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-17::naa.5000C500DC0D6D66:
tslsenclslot:slot:0:210235A3THH219000533-F19:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-18:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-18::naa.5000C500DC4A7B8D:
tslsenclslot:slot:0:210235A3THH219000533-F20:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-19:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-19::naa.5000C500DC4A80F8:
tslsenclslot:slot:0:210235A3THH219000533-F21:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-20:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-20::naa.5000C500DC476247:
tslsenclslot:slot:0:210235A3THH219000533-F22:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-21:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-21::naa.5000C500DC4922B4:
tslsenclslot:slot:0:210235A3THH219000533-F23:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-22:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-22::naa.5000C500DC47EF97:
tslsenclslot:slot:0:210235A3THH219000533-F24:AVAGO_MegaRAID:SKA3271883:lmr!SKA3271883!SLOT!0-1-23:0x01001100:1:-:-:-:-:-:LC=10,NX=1,SC=10,ID=0-1-23::naa.5000C500DC46F6CA:
[root@ECE-A03 gems]#
then we can start ECE cluster. And now the pdisk location could be reported correctly:
[root@ECE-A03 ~]# mmvdisk pdisk list --rg ece_rg1 --pdisk n003p021 -L
pdisk:
replacementPriority = 1000
name = "n003p021"
device = "//ECE-A03/dev/sdx"
recoveryGroup = "ece_rg1"
declusteredArray = "DA1"
state = "ok"
internalState = 00000.000
capacity = 15994458210304
freeSpace = 283467841536
fru = "ST16000NM001G-2"
location = "210235A3THH219000533-F24"
WWN = "naa.5000C500DC46F6CA"
server = "ECE-A06"
reads = 610
writes = 2952
bytesReadInGiB = 0.149
bytesWrittenInGiB = 0.721
IOErrors = 0
IOTimeouts = 0
mediaErrors = 0
checksumErrors = 0
pathErrors = 0
relativePerformance = 1.000
dataBadness = 0.000
rgIndex = 200
userLocation = ""
hardware = "ATA ST16000NM001G-2K SB30 ZL2LKR2Z"
hardwareType = Rotating 7200
nPaths = 1 active 1 total
nsdFormatVersion = 2
paxosAreaOffset = 16000896335872
paxosAreaSize = 4194304
logicalBlockSize = 512
ssdEndurancePercentage =
...
Refer to this link for more information: https://www.ibm.com/docs/en/spectrum-scale-ece/5.1.2?topic=location-mapping-lmr-disk