The
Elastic Storage Server (ESS) is a high-performance, GPFS™ network shared disk (NSD) solution that is made up of one or more building blocks. To work with ESS software, you need to have a working knowledge of Power Systems servers,
IBM Spectrum Scale™, and xCAT.
Power systems can run either in the BE (Big Endian) or the LE (Little Endian) mode. ESS is a building block consisting of 3 Power nodes and 1, 2, 4 or 6 disk enclosures. One Power node acts as the EMS (management node) node and other two Power nodes act as I/O server nodes (NSD nodes).
ESS PPCBE systems have a hardware management console (HMC), or any power system in BE mode, in place to manage the lifecycle activities of the node such as firmware upgrade, operating system re-image, power on or power off the LPARs, etc. HMC hides the complexities of the service network from the user. When the user tries to access network interfaces on EMS or I/O server nodes’ host operating system, the service network is not visible to the user and it is managed and serviced by the HMC node. In ESS PPCBE systems, HMC uses proprietary protocol to issue the lifecycle commands to nodes.
In ESS PPCLE systems, there is no HMC node. In these systems, lifecycle management of the Power nodes is done through IPMI protocol on host nodes and it requires physical service network interface with FSP network IP address, visible to the host operating system, to communicate with the FSP network.
The following networks are required for ESS systems:
- Service network: This network connects the flexible service processor (FSP) on the management server and I/O server nodes as shown in blue in the following ESS PPCBE networking diagram. On ESS PPCBE, the HMC runs the Dynamic Host Configuration Protocol (DHCP) server on this network. If the HMC is not included in the solution order, a customer-supplied HMC is used. On ESS PPCLE, since there is no HMC node, the EMS node runs the DHCP server on this network. The EMS FSP DHCP discovers the FSP nodes and assigns the dynamic IP. Once the IP has been assigned, it is converted to static IP and the EMS FSP DHCP server is brought down for the management network DHCP server to start working for xCAT management.
- Management and provisioning network: This network connects the management server to the I/O server nodes (and HMCs, depending on the platform) as shown as yellow in following networking diagrams. The management server runs DHCP on the management and the provisioning network. If a management server is not included in the solution order, a customer-supplied management server is used.
- Clustering network: This high-speed network is used for clustering and client node access. It can be a 10 Gigabit Ethernet (GbE), a 40 GbE, or an InfiniBand network. It might not be included in the solution order.
- External and campus management network: This public network is used for external and campus management of the management server, the HMC (if applicable, depending on the platform), or both.
Networking diagram of ESS PPCBE system
Networking diagram of ESS PPCLE system
Network interfaces available on ESS 5.x PPCBE on EMS Node
[root@ems2 ~]# ifconfig
bond0: flags=5187 mtu 1500
inet 11.1.202.20 netmask 255.255.255.0 broadcast 11.1.202.255
ether f4:52:14:74:97:d0 txqueuelen 0 (Ethernet)
RX packets 90966944233 bytes 134068570178083 (121.9 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22981099384 bytes 29508652303179 (26.8 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP4p1s0: flags=6211 mtu 1500
ether f4:52:14:74:97:d0 txqueuelen 1000 (Ethernet)
RX packets 90966694893 bytes 134068552274198 (121.9 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22981083405 bytes 29508650257867 (26.8 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP4p1s0d1: flags=6211 mtu 1500
ether f4:52:14:74:97:d0 txqueuelen 1000 (Ethernet)
RX packets 255800 bytes 18367325 (17.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15972 bytes 2044416 (1.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f0: flags=4163 mtu 1500
inet 192.168.202.20 netmask 255.255.0.0 broadcast 192.168.255.255
ether 98:be:94:03:ef:70 txqueuelen 1000 (Ethernet)
RX packets 37883353 bytes 4643902147 (4.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4161858 bytes 6046508275 (5.6 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f1: flags=4163 mtu 1500
ether 98:be:94:03:ef:71 txqueuelen 1000 (Ethernet)
RX packets 385672 bytes 28844417 (27.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f2: flags=4099 mtu 1500
ether 98:be:94:03:ef:72 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f3: flags=4099 mtu 1500
ether 98:be:94:03:ef:73 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0 (Local Loopback)
RX packets 11666838 bytes 3413208752 (3.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11666838 bytes 3413208752 (3.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
In the preceding command output, there are two active networks visible. The enP7p128s0f0 network is used for the management of the cluster and bond0 is a high-speed network used for the data transmission. One interesting thing to note here is that the FSP service network is not visible to the host operating system. However, it has been hidden and it is running between HMC and node. The FSP network is hidden here as the ESS cluster is running in PPCBE mode.
Network interfaces available in ESS 5.x on PPCLE on EMS Node
[root@ems1 ~]# ifconfig
bond0: flags=5187 mtu 1500
inet 11.1.202.12 netmask 255.255.255.0 broadcast 11.1.202.255
ether f4:52:14:fd:58:9f txqueuelen 0 (Ethernet)
RX packets 34254230426 bytes 19101594194487 (17.3 TiB)
RX errors 0 dropped 2715 overruns 2715 frame 0
TX packets 163535035943 bytes 245892493130714 (223.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP4p1s0: flags=6211 mtu 1500
ether f4:52:14:fd:58:9f txqueuelen 1000 (Ethernet)
RX packets 265795 bytes 19059043 (18.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16338 bytes 2091264 (1.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP4p1s0d1: flags=6211 mtu 1500
ether f4:52:14:fd:58:9f txqueuelen 1000 (Ethernet)
RX packets 34253971236 bytes 19101575656512 (17.3 TiB)
RX errors 0 dropped 2715 overruns 2715 frame 0
TX packets 163535052859 bytes 245892540563838 (223.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f0: flags=4163 mtu 1500
inet 192.168.202.12 netmask 255.255.0.0 broadcast 192.168.255.255
ether 98:be:94:00:59:00 txqueuelen 1000 (Ethernet)
RX packets 37281969 bytes 4598087405 (4.2 GiB)
RX errors 0 dropped 16 overruns 0 frame 0
TX packets 156651 bytes 77416070 (73.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f1: flags=4163 mtu 1500
inet 172.16.0.1 netmask 255.255.0.0 broadcast 172.16.255.255
ether 98:be:94:00:59:01 txqueuelen 1000 (Ethernet)
RX packets 660534 bytes 63162136 (60.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 368601 bytes 38308679 (36.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f2: flags=4099 mtu 1500
ether 98:be:94:00:59:02 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enP7p128s0f3: flags=4163 mtu 1500
ether 98:be:94:00:59:03 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 65745964 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0 (Local Loopback)
RX packets 26710999 bytes 34805802175 (32.4 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26710999 bytes 34805802175 (32.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
In the preceding command output, there are three active networks visible. The enP7p128s0f0 network is used for the management of the cluster and bond0 is a high-speed network used for the data transmission. One interesting thing to note here is that the FSP service network is visible. As there is no HMC in place in PPCLE, all the lifecycle operations of the node are now the responsibility of the EMS node. That is the reason, another network interface named enP7p128s0f1 is used here. The enP7p128s0f1 network is an FSP service network and it is used for servicing the node. This interface communicates with the FSP IPs assigned by FSP DHCP server to all the I/O server nodes including the EMS node. For example, following are the FSP IPs assigned by the FSP DHCP server:
[CMD_RESP]: node-8247-21l-212867a:
[CMD_RESP]: objtype=node
[CMD_RESP]: groups=all
[CMD_RESP]: bmc=172.16.0.9 FSP IP for EMS node.
[CMD_RESP]: cons=ipmi
[CMD_RESP]: mgt=ipmi
[CMD_RESP]: mtm=8247-21L
[CMD_RESP]: serial=212867A
[CMD_RESP]: nodetype=mp
[CMD_RESP]: hwtype=bmc
[CMD_RESP]:
[CMD_RESP]: node-8247-22l-212869a:
[CMD_RESP]: objtype=node
[CMD_RESP]: groups=all
[CMD_RESP]: bmc=172.16.0.10 FSP IP for IO Node 1
[CMD_RESP]: cons=ipmi
[CMD_RESP]: mgt=ipmi
[CMD_RESP]: mtm=8247-22L
[CMD_RESP]: serial=212869A
[CMD_RESP]: nodetype=mp
[CMD_RESP]: hwtype=bmc
[CMD_RESP]:
[CMD_RESP]: node-8247-22l-212868a:
[CMD_RESP]: objtype=node
[CMD_RESP]: groups=all
[CMD_RESP]: bmc=172.16.0.12 FSP IP for IO Node 2
[CMD_RESP]: cons=ipmi
[CMD_RESP]: mgt=ipmi
[CMD_RESP]: mtm=8247-22L
[CMD_RESP]: serial=212868A
[CMD_RESP]: nodetype=mp
[CMD_RESP]: hwtype=bmc
[CMD_RESP]:
[CMD_RESP]: RC: 0
These FSP IPs can be used to access the ASMI (Advanced System Management Interface). So in a nutshell, it can be stated that in case of PPCLE, all the HMC tasks have been offloaded to the EMS node using the IPMI protocol.
On the same path, IO nodes will never have the physical FSP interface visible. All the life cycle of the IO nodes will be managed by the EMS node either using via HMC propriety protocol (in case of PPCBE mode) or using IMPI command (in case of PPCLE mode).
Annexurea) Service network: This network connects the flexible service processor (FSP) on the management server and I/O server nodes.
b) xCAT (Extreme Cloud Administration Toolkit) is open-source distributed computing management software developed by IBM, used for the deployment and administration of Linux or AIX based clusters. It can be used to perform a variety of tasks that include:
1. Create and manage diskless clusters.
2. Install and manage many Linux cluster machines (physical or virtual) in parallel
3. Set up a high-performance computing software stack, including software for batch job submission, parallel libraries, and other software that is useful on a cluster
4. Cloning and imaging Linux and Windows machines
xCAT has specific features designed to take advantage of IBM hardware including:
1. Remote Power Control
2. Remote POST/BIOS console
3. Serial over LAN functions
4. Hardware alerts and vitals provided via SNMP and email
5. Inventory and hardware management
#IBMSpectrumScale#Softwaredefinedstorage#SpectrumScale