File and Object Storage

Introduction

IBM Storage Scale System is a pre-configured, high-performance storage appliance that integrates IBM's Storage Scale (formerly Spectrum Scale) software with purpose-built hardware to deliver a scalable, turnkey solution for large-scale data environments.  IBM Storage Scale System is designed to handle demanding workloads like AI, big data analytics, high-performance computing (HPC), and cloud-native applications. It combines advanced storage management software with optimized hardware to provide high throughput, scalability, and data reliability.

IBM Storage Scale System container-based offline upgrade workflow to version 6.2.1.0:

Upgrade can be done by using the following methods:

• Offline upgrade: This method is faster than an online update. The nodes are upgraded in parallel but the system is typically taken down for a period. You can update all nodes at the same time or a subset of them. Preferred when no cluster exists in the environment, or during a scheduled outage for the system.
• Online upgrade: This method requires a cluster and file systems, or multiple clusters and file systems. The cluster stays fully available. The chosen logic determines how many nodes to upgrade at the same time, usually one per building block to keep the recovery group accessible all the time. Also the quorum is maintained automatically during online upgrade.

In the following diagram we can see the general workflow that we have to follow when we perform an offline version of a system from a POWER EMS

Step by step upgrade process:

Now we are going to show the step by step and commands that we need to perform an upgrade:

1. Copy to the P9 Node management server the IBM Storage Scale System release image that you previously downloaded from IBM Fix Central. Preferably, copy the release image to the /home/deploy directory in the P9 Node management server.

   Scale_System_DAE_UNIFIED-6.2.1.0-ppc64LE-EMS

2. If there is a protocol node, copy to the P9 node management server the IBM Storage Scale installation toolkit package release image that you previously downloaded fromIBM Fix Central.

   Storage_Scale_Data_Management-5.2.1.0-ppc64LE-Linux

3. . Preferably, copy the release image to the /home/deploy directory in the P9 Node management server.
4. Set up the target release container in the Power management server by following the next steps

   • cd /home/deploy 
     tar -xvf Scale_System_DAE_UNIFIED-6.2.1.0-ppc64LE-EMS.tgz 
     xz --decompress ess_6.2.1.0_0827-21_dme_x86_64.tar.xz
     tar -xvf ess_6.2.1.0_0827-21_dme_x86_64.tar
     sh ess_6.2.1.0_0827-21_dme_x86_64

   • When the container prompt is displayed, run the essrun -N <management hostnames> config load command, as follows:

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ #
     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N ems,essio1,essio2,prt01,prt02 config load -p

   • Run the update precheck command to anticipate possible breakage during the update execution

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N ems,essio1,essio2,prt01,prt02 update --precheck

5. Update the management servers. NOTE: Complete the following steps for systems with one management server in the environment, for information about other scenarios please consulte de deployment guide
   • Run the first update

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N ems update --offline

   • If you encounter a message similar to the next one, follow its instructions: exit the container, reboot the management server, and restart the container. If you don't see this message, skip this step.

     - "Seems that kernel has changed. This will require a reboot "
           - "Please exit container and reboot {{ name }}"
           - "Restart container (./essmkyml --restart) once {{ name }} is back and run update again."

     
     

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # exit root@emsHostname:/ # reboot

   • When the EMS comes back, restart the container

     cd /home/deploy/ess_6.2.x.x_...dir/ 
     ./essmkyml --restart

   • Run a second update for the EMS

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ #essrun -N ems update --offline

   • If you encounter a message similar to the next one, follow its instructions: exit the container, reboot the management server, and run the reloadEms.sh script. If you don't see this message, skip this step.

     msg:
     - Please reboot the EMS node since OFED was updated or reinstalled.
     - YOU MUST RUN THE FOLLOWING SCRIPT AFTER REBOOT TO SET NETWORKING PARAMETERS AND ACTIVATE GPFS
     - 'Run this script: ''/opt/ibm/ess/tools/samples/reloadEms.sh'' when node emsHostname is back.'

     
     

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # exit
     root@emsHostname:/ # reboot

   • When the EMS comes back from reboot, execute the reloadEMS.sh script to set networking parameters and activate GPFS (Storage Scale). 

     /opt/ibm/ess/tools/samples/reloadEms.sh

   • If a cluster exists in the environment, restart the performance monitoring services and the IBM Storage Scale GUI

     systemctl restart pmsensors 
     systemctl restart gpfsgui

   • Update the system firmware only if need it.

     cd /install/ess/otherpkgs/rhels8/ppc64le/firmware/
     update_flash -v -f 01VL950_131_045.img
     update_flash -f 01VL950_131_045.img
     update_flash -c

   • Run a health check in EMS.

     essinstallcheck

6. Update I/O nodes
   • Start the update on all the I/O nodes.

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N essio1,essio2,essioN… update --offline

   • Restart the performance monitoring service on each upgraded I/O node.

     systemctl restart pmsensors

   • If a cluster exists in the environment, update the firmwares of the I/O nodes that were upgraded in the previous steps only if need it. For each of the I/O nodes access the node via SSH and update the enclosures and drives firmwares as follows:

     mmchfirmware --type storage-enclosure --fast-offline
     mmchfirmware --type drive --fast-offline

   • Update the system firmware if it needs. NOTE: Just for POWER systems

     cd /install/ess/otherpkgs/rhels8/ppc64le/firmware/
     update_flash -v -f 01VL950_131_045.img
     update_flash -f 01VL950_131_045.img
     update_flash -c

   • Run a health check in each of the I/O nodes that were upgraded

     essinstallcheck

7. Update the protocol nodes (If you have in your system)
   • Upgrade IBM Storage Scale in the protocol nodes. For more information about how to upgrade IBM Storage Scale in the protocol nodes, see  Performing offline upgrade or excluding nodes from upgrade by using installation toolkit in the IBM Storage Scale documentation.
   • Upgrade the operative system and other components by using the IBM Storage Scale System container.

     ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N prt1,prt2 update --offline

   • Update the system firmware only if need it. NOTE: Just for POWER protocol nodes.

     cd /install/ess/otherpkgs/rhels8/ppc64le/firmware/
     update_flash -v -f 01VL950_131_045.img
     update_flash -f 01VL950_131_045.img
     update_flash -c

   • After upgrade is completed ssh to each updated protocol node and restart GPFS (Storage Scale) services and CES services

     mmstartup 
     mmces node resume --start

   • Finally run a health check in each of the protocol nodes that were upgraded

     essinstallcheck

Conclusion

Updating an IBM Storage Scale System is straightforward, as demonstrated in this blog, largely due to the implementation of containers. This approach enables users to follow a more structured and streamlined workflow for performing updates using the essrun commands.

For more details on the upgrade process, as well as information on upgrading systems with different configurations and scenarios, please refer to the deployment guide available at the following link. IBM Storage Scale System Deployment Guide 6.2.1