Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

Hi T,
What type of zone are you using, port zone or WWPN? 

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

Hi T.,

can you please check the output of "lsfabric -node <node-id>" regarding the number of logins from the partner nodecanister.

Compare the output with the result you would expect based on the zoning.

According your information regarding the usage of 2 ports per fabric (per nodecanister) you should only see 8 logins on each nodecanister.

But message "1801 A node has received too many Fibre Channel logins from another node." is indicating a number of more then 16 logins.

Regards, Uwe

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

Hi T.,

can you please check the output of "lsfabric -node <node-id>" regarding the number of logins from the partner nodecanister.

Compare the output with the result you would expect based on the zoning.

According your information regarding the usage of 2 ports per fabric (per nodecanister) you should only see 8 logins on each nodecanister.

But message "1801 A node has received too many Fibre Channel logins from another node." is indicating a number of more then 16 logins.

Regards, Uwe

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..

Hi T.,

I suspect that the recommendation to use all ports is a relic from the old days, when systems could only support a few ports per controller.
Using two instead of eight ports obviously results in a loss of redundancy.
However, it must also be noted that inter-node communication is "only" used as a backup for communication via PCI Link over the midplane.

I've implemented dozens of Storwize and Flashsystem systems over the past 15 years. In that time, I've only encountered one system (V7000 Gen1) where the midplane was defective and had to be replaced.

Regards, Uwe

Hi T.,

can you please check the output of "lsfabric -node <node-id>" regarding the number of logins from the partner nodecanister.

Compare the output with the result you would expect based on the zoning.

According your information regarding the usage of 2 ports per fabric (per nodecanister) you should only see 8 logins on each nodecanister.

But message "1801 A node has received too many Fibre Channel logins from another node." is indicating a number of more then 16 logins.

Regards, Uwe

I still have a question about internode zoning in a flashsystem.

The documentation states:
https://www.ibm.com/docs/en/flashsystem-7x00/8.6.0?topic=details-san-configuration-zoning-rules-summary

If the system has only one control enclosure and one or more Fibre Channel ports that are attached to a SAN fabric, allow communication between node canisters over the Fibre Channel network. Creating one zone per system in each SAN fabric and adding all ports from this system into that zone satisfies this best practice.

The above is also told again in:
https://www.ibm.com/docs/en/flashsystem-9x00/8.7.0?topic=problem-control-enclosure-is-not-detected
If your system uses Fibre Channel, ensure that all cables are connected and zoning is set up according to the zoning rules. There must be a zone that includes all ports from all node canisters.

Now we have created a zone on both fabrics with all FC ports (as indicated in the documentation) and then we get the message
1801 A node has received too many Fibre Channel logins from another node.

If we use two ports per fabric, the 1801 error disappears.

So, how should I interpret the documentation?

Thanks

Hello Team,

We are about to implement a new FS7300, and internally, we have a long-standing discussion about inter-node zoning.

Andrew Martin blogged about this in the past. https://barrywhytestorage.blog/2019/11/05/using-dedicated-ports-for-inter-node-communication-and-replication/

Then it was written:

If you have a Storwize system, All communication between node canisters in the same IO group will occur across the PCI link on the midplane. If the PCI midplane fails, a Fibre Channel network will be used as a backup link.

Should I do something special for this inter-node communication?

If you have a single IO group Storwize system, then you don't need to do anything more. However it is a good idea to create a zone in your fabric that allows any FC ports from node canister 1 to communicate with node canister 2, just in case there is a PCI failure in the midplane link.

The 8.7.0 documentation indicates that for an FS7300 with 12 FC ports, 4 ports dedicated to inter-cluster communication are required (suggested).

Planning for more than four fabric ports per node canister - IBM Documentation

Now I'm curious if what applied to Storwize back then still applies to current Flash systems like the FS7300\FS5300?

We prefer to use the 4 FC ports for host traffic to the SVC and apparently these are only used as a backup because initially everything goes via midplane.

Thanks..