IBM FlashSystem

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.

Hi,
Use 'lsnodecanister'  when logged into the Cluster IP address.

you should see you have an offline node.  take note of the very left number in the output line, which is the ID assigned to that offline node,
then issue 'rmnodecanister x' whee x is the ID you took note of in the previous output.

you can now check 'lsnodecanister' again and you should only have the one node that is online.
then from the Service Assistant "http://<cluster IP address>/service
 Select the Candidate Node and from the pulldown menu select 'reboot' and click Go.
 *** Ensure you do not black popups, if you get a popup blocked notification,
 enable the popups, and redo the Reboot request.

Once the node reboots it should automatically add to your cluster.

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.

Thank you for responding, Glen.

Below is the output of lsnodecanister:

IBM_Storwize:SAN02P04:superuser>lsnodecanister

id name  UPS_serial_number WWNN             status IO_group_id IO_group_name config_node UPS_unique_id hardware iscsi_name                              iscsi_alias panel_name enclosure_id canister_id enclosure_serial_number site_id site_name

3  node1                   500507680D005DBC online 0           io_grp0       yes                       T5L      iqn.1986-03.com.ibm:2145.san02p04.node1             01-1       1            1           781C582        

Only one shows up.

I've tried installing several canisters in the 2nd slot with the original RAM and batter - they all come online, show the correct serial and WWNN numbers but the canisters don't see each other with error 734 1 1 0. The system does not show anything connected to the 2nd slot with any canister I try. I have been able to login to the 2nd canister and even install matching code version with still no communication between the system and the 2nd canister. Does this mean there's possibly an issue with the mid-plane of the chassis that connects interconnects the two canisters?

Hi,
Use 'lsnodecanister'  when logged into the Cluster IP address.

you should see you have an offline node.  take note of the very left number in the output line, which is the ID assigned to that offline node,
then issue 'rmnodecanister x' whee x is the ID you took note of in the previous output.

you can now check 'lsnodecanister' again and you should only have the one node that is online.
then from the Service Assistant "http://<cluster IP address>/service
 Select the Candidate Node and from the pulldown menu select 'reboot' and click Go.
 *** Ensure you do not black popups, if you get a popup blocked notification,
 enable the popups, and redo the Reboot request.

Once the node reboots it should automatically add to your cluster.

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.

Hello Edstrom IT,
  indeed the '734' error states that the PCIe link is not communicating and therefore cannot see the other node.
While it is possible to be caused by midplane, it is more likely that the Active Canister is at fault.

If this v5010 has FibreChannel interfaces, I suggest connecting 2 ports per node to a switch and zone them to see each other all in 1 zone.
The nodes will communicate via the FiberChannel paths and the replaced node can be added to the cluster that way.
Once you have a 2 node cluster, then try rebooting / reseating (or replacing) node -1 to resolve the '734' problem.

Thank you for responding, Glen.

Below is the output of lsnodecanister:

IBM_Storwize:SAN02P04:superuser>lsnodecanister

id name  UPS_serial_number WWNN             status IO_group_id IO_group_name config_node UPS_unique_id hardware iscsi_name                              iscsi_alias panel_name enclosure_id canister_id enclosure_serial_number site_id site_name

3  node1                   500507680D005DBC online 0           io_grp0       yes                       T5L      iqn.1986-03.com.ibm:2145.san02p04.node1             01-1       1            1           781C582        

Only one shows up.

I've tried installing several canisters in the 2nd slot with the original RAM and batter - they all come online, show the correct serial and WWNN numbers but the canisters don't see each other with error 734 1 1 0. The system does not show anything connected to the 2nd slot with any canister I try. I have been able to login to the 2nd canister and even install matching code version with still no communication between the system and the 2nd canister. Does this mean there's possibly an issue with the mid-plane of the chassis that connects interconnects the two canisters?

Hi,
Use 'lsnodecanister'  when logged into the Cluster IP address.

you should see you have an offline node.  take note of the very left number in the output line, which is the ID assigned to that offline node,
then issue 'rmnodecanister x' whee x is the ID you took note of in the previous output.

you can now check 'lsnodecanister' again and you should only have the one node that is online.
then from the Service Assistant "http://<cluster IP address>/service
 Select the Candidate Node and from the pulldown menu select 'reboot' and click Go.
 *** Ensure you do not black popups, if you get a popup blocked notification,
 enable the popups, and redo the Reboot request.

Once the node reboots it should automatically add to your cluster.

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.

These canisters don't have fiberchannel but they have SFP+

There's one free sfp+ on both canisters. Is it possible to do this without fiberchannel?

Hello Edstrom IT,
  indeed the '734' error states that the PCIe link is not communicating and therefore cannot see the other node.
While it is possible to be caused by midplane, it is more likely that the Active Canister is at fault.

If this v5010 has FibreChannel interfaces, I suggest connecting 2 ports per node to a switch and zone them to see each other all in 1 zone.
The nodes will communicate via the FiberChannel paths and the replaced node can be added to the cluster that way.
Once you have a 2 node cluster, then try rebooting / reseating (or replacing) node -1 to resolve the '734' problem.

Thank you for responding, Glen.

Below is the output of lsnodecanister:

IBM_Storwize:SAN02P04:superuser>lsnodecanister

id name  UPS_serial_number WWNN             status IO_group_id IO_group_name config_node UPS_unique_id hardware iscsi_name                              iscsi_alias panel_name enclosure_id canister_id enclosure_serial_number site_id site_name

3  node1                   500507680D005DBC online 0           io_grp0       yes                       T5L      iqn.1986-03.com.ibm:2145.san02p04.node1             01-1       1            1           781C582        

Only one shows up.

I've tried installing several canisters in the 2nd slot with the original RAM and batter - they all come online, show the correct serial and WWNN numbers but the canisters don't see each other with error 734 1 1 0. The system does not show anything connected to the 2nd slot with any canister I try. I have been able to login to the 2nd canister and even install matching code version with still no communication between the system and the 2nd canister. Does this mean there's possibly an issue with the mid-plane of the chassis that connects interconnects the two canisters?

Hi,
Use 'lsnodecanister'  when logged into the Cluster IP address.

you should see you have an offline node.  take note of the very left number in the output line, which is the ID assigned to that offline node,
then issue 'rmnodecanister x' whee x is the ID you took note of in the previous output.

you can now check 'lsnodecanister' again and you should only have the one node that is online.
then from the Service Assistant "http://<cluster IP address>/service
 Select the Candidate Node and from the pulldown menu select 'reboot' and click Go.
 *** Ensure you do not black popups, if you get a popup blocked notification,
 enable the popups, and redo the Reboot request.

Once the node reboots it should automatically add to your cluster.

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.

Hi Edstrom,
 an "SFP+" is merely a tranciever for an interface of either 10GbE  (Ethernet) or FC (Fibre Channel).
Do the other interfaces connect directly to Hosts or via a switch,
 and is that Switch an Ethernet Switch (LAN) or a Fibre Channel (SAN) switch ?

If the later, then you can zone both the nodes interfaces to see each other on the SAN switch(es).
However if you are Ethernet, unfortunately this method will not work.

You would need an outage, and given the state of things, the storage likely will not come back up from a power cycle.
 So I recommend opening a case with IBM to assist you further.

These canisters don't have fiberchannel but they have SFP+

There's one free sfp+ on both canisters. Is it possible to do this without fiberchannel?

Hello Edstrom IT,
  indeed the '734' error states that the PCIe link is not communicating and therefore cannot see the other node.
While it is possible to be caused by midplane, it is more likely that the Active Canister is at fault.

If this v5010 has FibreChannel interfaces, I suggest connecting 2 ports per node to a switch and zone them to see each other all in 1 zone.
The nodes will communicate via the FiberChannel paths and the replaced node can be added to the cluster that way.
Once you have a 2 node cluster, then try rebooting / reseating (or replacing) node -1 to resolve the '734' problem.

Thank you for responding, Glen.

Below is the output of lsnodecanister:

IBM_Storwize:SAN02P04:superuser>lsnodecanister

id name  UPS_serial_number WWNN             status IO_group_id IO_group_name config_node UPS_unique_id hardware iscsi_name                              iscsi_alias panel_name enclosure_id canister_id enclosure_serial_number site_id site_name

3  node1                   500507680D005DBC online 0           io_grp0       yes                       T5L      iqn.1986-03.com.ibm:2145.san02p04.node1             01-1       1            1           781C582        

Only one shows up.

I've tried installing several canisters in the 2nd slot with the original RAM and batter - they all come online, show the correct serial and WWNN numbers but the canisters don't see each other with error 734 1 1 0. The system does not show anything connected to the 2nd slot with any canister I try. I have been able to login to the 2nd canister and even install matching code version with still no communication between the system and the 2nd canister. Does this mean there's possibly an issue with the mid-plane of the chassis that connects interconnects the two canisters?

Hi,
Use 'lsnodecanister'  when logged into the Cluster IP address.

you should see you have an offline node.  take note of the very left number in the output line, which is the ID assigned to that offline node,
then issue 'rmnodecanister x' whee x is the ID you took note of in the previous output.

you can now check 'lsnodecanister' again and you should only have the one node that is online.
then from the Service Assistant "http://<cluster IP address>/service
 Select the Candidate Node and from the pulldown menu select 'reboot' and click Go.
 *** Ensure you do not black popups, if you get a popup blocked notification,
 enable the popups, and redo the Reboot request.

Once the node reboots it should automatically add to your cluster.

a canister node failed about a month ago and we removed it from the system.

we replaced the node with another v5010 PN: 01ac370. We put the original RAM and cache battery in place to retain the service IP from old canister to new canister. 

I can log in to the new canister, I was able to remove the system info it previously contained and now shows up with the correct WWNN numbers and serial. 

The active node however does not see the new node which is right now in "candidate" mode. When i run svcinfo lsnodecandidate there's nothing in return.

Active node information:

Here is info on the new canister:

Am i missing something? thank you in advance.