Virtual LANs (VLANs) are often associated with secure networks because they provide a means of separating network devices onto independent networks. Though the physical network infrastructure is shared, unicast, multicast, and broadcast traffic from a network device in a VLAN is restricted to other devices within that same VLAN.
When planning for IBM Spectrum Scale CES Protocols, consider the following:
- How many networks must have access to data presented by SMB, NFS, and Object protocols?
- How many network cables will connect to each protocol node and will each cable allow access to a single network or multiple networks?
- Will all protocol nodes see the same networks or will CES Groups be needed in order to group protocol nodes by their network visibility?
- Plan authentication such that all protocol nodes can see the authentication server(s) as well as all client nodes that you plan on joining to the authentication domain(s).
- Forward DNS lookup must be in place for all CES-IPs
- Reverse DNS lookup must be in place for all CES-IPs
- CES-IPs cannot already be in use on the network
- CES-IPs must match to a subnet/route existing on all protocol nodes (an exception to this is when CES groups are in place)
What exactly is a CES-IP?
Cluster Export Services (CES) is a functionality within IBM Spectrum Scale enabling SMB, NFS, and Object protocols. No matter which protocol(s) you choose, all are accessible similarly, via a floating pool of IPs called CES-IPs. This pool of CES-IPs is considered floating because each IP can move independently among all protocol nodes. In the event of protocol node failure, accessibility to all protocols is maintained as the CES-IPs automatically move from the failed protocol node to a healthy protocol node.
How are CES-IPs assigned?
CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. See the following example of aliased CES IPs in a flat network environment or a single VLANd environment in which the switch ports are set to Access mode and thus, do not need VLAN tagging.
Example of aliased CES IPs via the ip addr command:
eth1:
Example of pre-existing routes:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1
10.11.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
In the above example, eth1 pre-exists with an established route and IP: 10.11.1.122. This is manually assigned and must be accessible prior to any CES configuration. Once CES services are active, CES IPs are then automatically aliased to this base adapter, thus creating eth1:0 and eth1:1. The floating CES IPs assigned to the aliases are 10.11.1.5 and 10.11.1.7. Both CES IPs are allowed to move to other nodes in case of failure. This automatic movement, combined with the ability to manually move CES IPs, may cause a variance in the number of aliases and CES IPs among protocol nodes. The data0 interface illustrates how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES-IPs.
Example distribution of CES-IPs among 2 protocol nodes after enablement of protocols. Our example protocol node from above is called protocol-node-1:
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.11.1.5 protocol-node-1 none none
10.11.1.6 protocol-node-2 none object_database_node,object_singleton_node
10.11.1.7 protocol-node-1 none none
10.11.1.8 protocol-node-2 none none
VLAN tags and CES-IPs
A network switch port can be considered a trunk port if it gives access to multiple VLANs.
When this occurs, it is necessary for a VLAN tag to be added to each frame. This VLAN tag is an identification allowing switch(es) to contain traffic within specific networks. If multiple networks must access data from IBM Spectrum Scale protocol nodes, then one possible option is to configure trunk ports on the switch directly connected to the Spectrum Scale Protocol nodes. Once a trunk port exists, VLAN tags will be necessary on the connected network adapters. Remember that CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. Due to this, the existence of VLAN tags require a pre-existing network adapter with an established route and IP so that CES-IPs can alias to it. See what this looks like below:
Example of aliased CES IPs via the ip addr command:
eth1:
Example of pre-existing routes:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1.3016
10.30.16.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1.3016
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
As in the no VLAN tag example, an existing network adapter must be present so that CES-IPs can alias to it. Notice the non-VLAN base adapter eth1, has no IPs assigned. In this example, our pre-existing network adapter with an established route and IP is eth1.3016. The IP for eth1.3016 is 10.30.16.122 and the VLAN tag is 3016. This pre-existing IP can be used for network verification prior to CES-IP configuration by simply pinging it from external to the cluster or pinging it from other protocol nodes. It is good practice to make sure all protocol node base adapter IPs are accessible before enabling protocols. The data0 interface illustrates how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES-IPs.
Example distribution of CES-IPs among 2 protocol nodes after enablement of protocols. Our example protocol node from above is called protocol-node-1:
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.30.16.5 protocol-node-1 none none
10.30.16.6 protocol-node-2 none object_database_node,object_singleton_node
10.30.16.7 protocol-node-1 none none
10.30.16.8 protocol-node-2 none none
Multiple VLAN tags and CES-IPs
As an added example, a node with 2 network adapters devoted to CES protocols: eth1 and eth2. eth1 has 2 VLANs associated: 3016 and 3017. eth2 has 1 VLAN associated: 80.
Example of aliased CES IPs via the ip addr command:
eth1:
Example of the pre-existing routes:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1.3016
10.30.16.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1.3016
10.30.17.0 0.0.0.0 255.255.255.0 U 400 0 0 eth1.3017
10.11.80.0 0.0.0.0 255.255.255.0 U 200 0 0 eth2.80
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
Example distribution of CES-IPs from multiple VLANs among 2 protocol nodes after enablement of protocols. Our example protocol node from above is called protocol-node-1:
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.11.80.54 protocol-node-2 none none
10.11.80.55 protocol-node-1 none none
10.30.16.5 protocol-node-1 none none
10.30.16.6 protocol-node-2 none none
10.30.16.7 protocol-node-1 none none
10.30.16.8 protocol-node-2 none none
10.30.17.100 protocol-node-1 none none
10.30.17.101 protocol-node-2 none none
10.30.17.102 protocol-node-2 none object_database_node,object_singleton_node
10.30.17.103 protocol-node-1 none none
References
#Softwaredefinedstorage#IBMStorageScale