Primary Storage

 View Only
Expand all | Collapse all

Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

  • 1.  Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Thu June 01, 2023 06:05 AM

    Hello,

    A node failed a while ago and then forced removed from the system. 

    Auditlog Entry 386
    Sequence Num : 2886
    Timestamp : Tue Jan 11 11:32:11 2022
    : Epoch + 1641886331
    Cluster User : ram
    Challenge : 
    SSH IP Address : 
    Result Code : 0
    Result Obj ID : 
    Action Cmd : svctask rmnode -force -gui 2
    Source_Panel : 
    Target_Panel : 


    We have replaced the node which is now seen in candidate state but it is not seen by the system, thus lsnodecandidate is empty.

    The new node is as follow:

    node_status Candidate
    node_product_mtm_copy 6535-HC4
    node_FRU_part 01AC370
    product_name IBM Storwize V5000
    node_code_version 8.2.1.11
    node_code_build 147.20.2007010940000

    The current node in the system is as follow:

    node_name node1
    node_status Active
    config_node Yes

    product_MTM 6535-HC4

    node_FRU_part 01LJ608
    product_name  Storage V3700 V2

    node_code_version 7.7.1.4
    node_code_build 130.16.1611221559000

    svcinfo lsnodecandidate -delim :

    svcinfo lsiogrp -delim :
    id:name:node_count:vdisk_count:host_count:site_id:site_name
    0:io_grp0:1:2:2:

    sainfo lsservicenodes
    panel_name cluster_id           cluster_name node_id node_name relation node_status error_data
    01-1       000000E020207ES2 V3700    1       node1     local    Active      

    svcinfo lsnode -delim :
    id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name:config_node:UPS_unique_id:hardware:iscsi_name:iscsi_alias:panel_name:enclosure_id:canister_id:enclosure_serial_number:site_id:site_name
    1:node1::500507680G00xxxx:online:0:io_grp0:yes::T5L:iqn.1986-03.com.ibm:2145.v3700.node1::01-1:1:1:781xxxx::

    Any idea why the nodes aren't seen each other ?

    Can the difference in the code be the issue ?

    or the fact that the new node is seen as V5000 instead of the V3700 ?

    Any suggestion on how to get around this ?

    Thank you



    ------------------------------
    Justine Uwase
    ------------------------------


  • 2.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Fri June 02, 2023 04:16 AM

    Update:

    We also tried with a new canister with node_FRU_part 01LJ608 and the issue remains ... 

    Thank you for your feedback



    ------------------------------
    Justine Uwase
    ------------------------------



  • 3.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Mon June 05, 2023 11:30 AM

    Hello Justine,  the Lenovo V3700 V2 is and IBM V5000 Gen2 Model... and in the Gen 2 family intermix of IBM and Lenovo parts will not work.
    The first node showed:
    product_name IBM Storwize V5000
    Did that node come from stock or was it used in another system before?  A node with IBM in product name will not join a 'Lenovo'cluster
    Please check as well the product_name for the other new node you  have mentioned?
    Fru is used for both IBM and Lenovo, but once a system was used in a cluster is becomes either 'IBM' or 'Lenovo'.  that why I asked if that was a reused node
    Sabine Gronert, SpecV support



    ------------------------------
    Sabine Gronert
    ------------------------------



  • 4.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Tue June 06, 2023 03:54 AM
    Hi Sabine,

    Thank you so much for your response ... I have asked for a new/unused part as the ones we got have been used and branded V5000.

    Have a wonderful day 





  • 5.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Fri June 02, 2023 08:57 AM

    Hello Justine,

    long time no hear ;)

    Supposedly the spare part is not considered suitable as candidate due to its code level of 8.2.1.11.
    You should attempt to go forward with a node rescue to downgrade the replacement node canister to the system's code level.
    Log on to SAGUI > Select the Replacement Node (CANDIDATE!) > Re-install Machine Code > Rescue from another node



    ------------------------------
    Christian Schroeder
    IBM SpecV Storage Support with Passion
    ------------------------------



  • 6.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Mon June 05, 2023 02:59 AM

    Hi Christian,

    Yes long time, ended ....

     I will try the Node rescue ... However, I I have little hope that it will work as the nodes do not see each other somehow ...

    Thank you,



    ------------------------------
    Justine Uwase
    ------------------------------



  • 7.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    User Group Leader
    Posted Mon June 05, 2023 05:30 AM

    Hi - 

    to clarify - the nodes DO see each other if they show up in sainfo lsservicenodes.  lscandidatenodes will only show nodes that are valid clustering partners, which in this case they are not, as one is installed with an OEM FRU build image and one has the IBM product image.  To correct this, you need to upgrade to the same code version as your cluster so they appear to be valid clustering partners.



    ------------------------------
    Evelyn Perez
    IBM Senior Technical Staff Member
    IBM Storage Virtualize Software Architect for SVC and FlashSystem
    ------------------------------



  • 8.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Mon June 05, 2023 09:18 AM
    Hi Evelyne,

    First, thank you so much for your response.

    We tried the Node rescue and it didn't work as Nodes don't see each other via lsservicenodes: Each node sees itself only ... For the new node, we have to use its service gui or via direct connection to get its information  ...

    I have obtained the old code version and we'll try to force downgrade the new node controller to 7.7 to see if this will allow the internode communication.

    Stay turned !

    Thank you again







  • 9.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Tue September 12, 2023 10:39 AM

    Hello.

    Have You downgraded the code version? What is the result? Do primary node see the second node as candidate?

    Any further steps? What was the result because I have similar issue.

    Best regards.



    ------------------------------
    Slawomir Bielawiec
    ------------------------------



  • 10.  RE: Help with V3700 : New node in candidate state but not seen in lsnodecandidate to be re-added.

    Posted Fri September 22, 2023 03:40 AM
    Hello,

    The issue was that one node was IBM and another Lenovo version. 

    We had to get the 2 that matches with a Lenovo machine type 69xx

    Have a great day.