This blog provides an introduction to IBM Spectrum Scale Erasure Code Edition and then describes a procedure that is followed for installing and configuring IBM Spectrum Scale Erasure Code Edition using the installation toolkit.
Introduction to IBM Spectrum Scale Erasure Code Edition:IBM Spectrum Scale Erasure Code Edition delivers valued IBM Spectrum Scale features and Network RAID to commodity network storage-rich servers which can contain NVMe, SSD, or HDD drives. The data and parity are evenly distributed across servers and storage devices to survive concurrent failure of servers and storage devices. Using software based declustered erasure codes, storage utilization is improved, and cost is reduced. Storage can be expanded by adding storage-rich servers. The Disk Hospital component of IBM Spectrum Scale Erasure Code Edition can identify failing disks for replacement and detect and correct silent data corruption.
IBM Spectrum Scale Erasure Code Edition Prerequisites:IBM Spectrum Scale Erasure Code Edition requires several software packages in addition to the base operating system.
Before installing IBM Spectrum Scale Erasure Code Edition, your network must pass the latency network KPIs for Ethernet networks.
Note: In the IBM Spectrum Scale Erasure Code Edition, customers will be required to meet the following network KPI metrics before an installation is completed.
Network requirements and precheck:This topic describes the networking requirements that must be met before using IBM Spectrum Scale Erasure Code Edition.
In the IBM Spectrum Scale Erasure Code Edition configuration, network bandwidth is consumed by the client workload as well as the backend Erasure Code traffic between nodes. For read I/O, every 1.0 Gbps of usable bandwidth requires 2.0 Gbps of total bandwidth. For write, the overhead depends on the selected Erasure Code. When writing with 8+3P, each 1.0 Gbps of usable bandwidth requires 2.4 Gbps of total bandwidth. This factor is 2.25 for 8+2P, 2.5 for 4+2P, and 2.75 for 4+3P.
Additional network considerations and requirements are as follows:• Linux bonding is supported on mode 1 (active-backup) for Ethernet and RDMA and mode 4 (IEEE 802.3ad) on Ethernet only. For mode 4 any xmit_hash_policy is supported, however, it is recommended to use layer3+4.
• Jumbo frames of 9000 MTU (on Ethernet) or higher (on RDMA) is recommended.
• A dedicated storage network for IBM Spectrum Scale Erasure Code Edition and daemon traffic is strongly recommended.
Network Key Performance Indicators are listed as follows:
• The average latency between any two storage nodes should be 1 msec or less.
• The maximum latency between any two storage nodes should be 2 msec or less.
• The standard deviation should be 0.333 msec or less on these measurements.
• These metrics should be collected over an extended period, at least 500 seconds for each measurement.
Note: You can use the KOET tool for any assistance with these measurements. Contact IBM for any further issues.
The following rpms are required to be installed:• sg_utils
• nmve-cli
• storcli (if using SAS drives with LSI HBA)
The above-mentioned rpms should be installed and configured on all nodes in the cluster.
Furthermore, it is important to ensure that you have the latest version of Mellanox OFED installed on each node. Likewise, the driver versions should be maintained at a consistent level across all nodes.
Note: /etc/hosts file should have a proper host entry in below mentioned order,
IP FQDN(Fully Qualified Domain Name) SHORTNAME
Password less ssh should be configured from admin node to all other nodes and to itself via IP, short name and FQDN.
IBM Spectrum Scale Erasure Code Edition Installer architecture overview:
Steps for installing IBM Spectrum Scale Erasure Code Edition:ECE cluster can have both scale-out node as well as non scale-out node.
IBM Spectrum Scale Erasure Code Edition is available in a separate install package and you can install it by using the installation toolkit.
Use the following steps to install IBM Spectrum Scale Erasure Code Edition.
1. Download the IBM Spectrum Scale Erasure Code Edition self-extracting package from the IBM Spectrum Scale page on Fix Central.2. Extract the install package. The installation toolkit gets extracted to/usr/lpp/mmfs/5.0.3.1/installer/ directory.
For example, refer to the following command,
./Spectrum_Scale_Erasure_Code-5.0.3.1-x86_64-Linux-install --text-only
3. Change the directory to the location where the installation toolkit is extracted to.cd /usr/lpp/mmfs/5.0.3.1/installer/
4. Specify the installer node and the setup type in the cluster definition file. The setup type must be ece for IBM Spectrum Scale Erasure Code Edition../spectrumscale setup -s InstallerNodeIP -st ece
NOTE: If you have a cluster already existing, you need to run this command before
executing step 5.
./spectrumscale config populate -N ScaleOutNodeIP
5. Add scale out nodes for IBM Spectrum Scale Erasure Code Edition in the cluster definition file../spectrumscale node add NodeName -so
Specify any other node designations in the cluster definition file.
You can use the following command to display the list of nodes specified in the
cluster definition file and the respective node designations.
./spectrumscale node list
6. Define the recovery group for IBM Spectrum Scale Erasure Code Edition in the cluster definition file../spectrumscale recoverygroup define -N Node1,Node2,...,NodeN
7. Perform environment prechecks before invoking the installation toolkit install command../spectrumscale install -pr
8. Perform the installation toolkit install operation../spectrumscale install
9. Check whether the current configuration has single declustered array or multiple declustered arrays by running the following command to check the DA information./usr/lpp/mmfs/bin/mmvdisk rg list --rg --da
To get the recovery group name use the following command:
./spectrumscale recoverygroup list
a. Single declustered array: ./spectrumscale vdiskset define -rg RgName -code RaidCode -bs BlockSize -ss SetSize
Use the following command for vdiskset name:
./spectrumscale vdiskset list
If you want only single vdiskset part of file system then use the following command:
./spectrumscale filesystem define -fs FileSystem -vs VdiskSet
If you need protocol node, execute step 10 first and then run the deploy command..
Perform environment prechecks before invoking the installation toolkit deploy command.
./spectrumscale deploy --pre
Perform the installation toolkit deploy operation.
./spectrumscale deploy
b. Multiple declustered array:/usr/lpp/mmfs/bin/mmvdisk vdiskset define --vdisk-set --recovery-group
--code --block-size --set-size
--declustered-array
Use the following command for vdiskset name
/usr/lpp/mmfs/bin/mmvdisk vdiskset list
Note: Run multiple declustered array command for each DA separately with unique
vdiskset name. For example, if you have three DAs run command for three times each for
one DA with unique vdiskset name.
Once all DAs have vdiskset defined run the following command to create vdisks. Use
the vdiskset name defined for creation.
/usr/lpp/mmfs/bin/mmvdisk vdiskset create --vdisk-set VdiskSet[,VdiskSet...
If you want multiple vdiskset part of filesystem use the following command:
/usr/lpp/mmfs/bin/mmvdisk filesystem create --file-system Name --vdisk-set Name,Name...
10. For protocol node configuration, use the following command. If you don’t want protocol node at this point of time, you can do this later.a. Assign cluster export service(CES) protocol service IP addresses.
These addresses are separate from the IP addresses are used internally
by the cluster../spectrumscale config protocols -e
b. Add nodes as a protocol node in the cluster definition file../spectrumscale node add NodeName -p
c. Enable required protocol (NFS, SMB or Object):./spectrumscale enable nfs | smb | object
d. Configure protocol cesSharedRoot file system:./spectrumscale config protocols -f cesSharedRoot -m /gpfs/cesSharedRoot
Where cesSharedRoot: File system name that can be
used for CES shared root that is needed
for protocol configuration.
/gpfs/cesSharedRoot : CES shared root filesystem
mount point.
IBM Spectrum Scale Erasure Code Edition installation using installation toolkit documented in the Knowledge Center : See the Installing IBM Spectrum Scale Erasure Code Edition section in the IBM Spectrum Scale Erasure Code Edition Knowledge Center.
#IBMSpectrumStorage#Flashstorage#IBMSpectrumScale#softwaredefinedstorage#ibmstorage#IBMSoftwareDefinedStorage#GPFS#Erasurecodeedition#Softwaredefinedinfrastructure#IBMSpectrumScaleErasureCodeEdition#Cloudstorage#Softwaredefinedstorage#IBMSpectrumScale