File and Object Storage

 View Only

IBM Storage Scale System upgrade process from Utility EMS

By Jesus Isaac Soriano Pineda posted 6 days ago

  

Introduction

IBM Storage Scale System is supported in different hardware, POWER, Utility Pollywell and some compatible x86 nodes (BYOEMS option). 

In this blog we will present the upgrade process of IBM Storage Scale System from another supported platform hardware, the Pollywell Utility hardware x86 based architecture. The main difference between this platform and the one presented previously in (link to the previous blog), is that in this platform the EMS is deployed in a VM that runs on the utility hardware, so for the upgrade process we must update the BareMetal node and the virtual machine (VM). 

IBM Storage Scale System container-based offline upgrade workflow to version 6.2.2.0 

Like the blog (IBM Storage Scale System upgrade process to version 6.2.1.0 in a POWER EMS) we have two upgrade scenarios 

  • Offline upgrade: This method is faster than an online update. The nodes are upgraded in parallel, but the system is typically taken down for a period of time. You can update all nodes at the same time or a subset of them. This method is preferred when no cluster exists in the environment, or during a scheduled outage for the system. 

  • Online upgrade: This method requires a cluster and file systems, or multiple clusters and file systems. The cluster stays fully available. The chosen logic determines how many nodes to upgrade at the same time, usually one per building block to keep the recovery group accessible all the time. Also, the quorum is maintained automatically during online upgrade. 

In the following diagram we can see the general workflow that we have to follow when we perform an offline version of a system from a Utility EMS. 

Step by step upgrade process 

  1. Copy to the Utility EMS Virtual Machine the IBM Storage Scale System release image that you previously downloaded from IBM Fix Central. Preferably, copy the release image to the /home/deploy directory in the EMS VM Node. 

Scale_System_DME_UNIFIED-6.2.2.0-x86_64-EMS

  1. If there is a protocol node, copy to the Utility EMS Virtual Machine node the IBM Storage Scale installation toolkit package release image that you previously downloaded fromIBM Fix Central.

Storage_Scale_Data_Management-5.2.2.0-x86_64-Linux

  1. Set up the target release container in the Utility EMS Virtual Machine by following the next steps. 

[root@emsvm ~]# cd /home/deploy 
[root@emsvm ~]# tar -xvf Scale_System_DME_UNIFIED-6.2.2.0-x86_64-EMS 
[root@emsvm ~]# xz –decompress ess_6.2.2.0_1210-21_dme_x86_64.tar.xz 
[root@emsvm ~]# tar -xvf ess_6.2.2.0_1210-21_dme_x86_64.tar 
[root@emsvm ~]# sh ess_6.2.2.0_1210-21_dme_x86_64

  • When the container prompt is displayed, run the essrun -N <management hostnames> config load command first for the Utility Bare Metal and then for the rest of the nodes, as follows: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N utilityBareMetal config load 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N emsvm,essio1,essio2,prt01,prt02 config load -p

  • Run the update precheck command to anticipate breakage during the update execution: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N utilityBareMetal,emsvm,essio1,essio2,prt01,prt02 update --precheck

  1. Update the management servers. NOTE: Complete the following steps for systems with one management server in the environment, for information about other scenarios please consult the deployment guide 

  • Update UtilityBareMetal node; from the container, run this command: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N utilityBareMetal update --offline

  • You may be prompted to reboot and run a second time if the kernel or OFED is changed. If you do not see the following message and the upgrade is successful skip to step 5. 

msg: 
    - "Seems that kernel has changed. This will require a reboot " 
    - "Please exit container, shutdown VM and reboot utilityBareMetal” 
    - "Restart VM (./emsvm --start-EMS ) and container (./essmkyml --restart) once utilityBareMetal is back and run update again."

  • Reboot the Utility node server with the following commands: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # exit  
[root@utilityBareMetal ~]# systemctl reboot

  • Once the server boots again validate that the VM is running: 

[root@utilityBareMetal ~]# virsh  list --state-running 
 Id   Name        State 
--------------------------- 
 1    EMSVM-23E   running

  • If the VM is not running, start it with the following commands: 

[root@utilityBareMetal ~]# cd /serv/EMSVM-main/ ; ./emsvm --start-EMS

  • Log into to the VM and restart the container: 

[root@emsvm ~]# cd /home/deploy/ess_6.2.x.x_...dir/ ; ./essmkyml --restart

  • Run update again: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N utilityBareMetal update --offline

  • At the end of the update, there should be a message indicating it was completed. 

msg: 
    - "Upgrade of ESS Utility Node (Bare metal) Complete."

  1. Update EMSVM node; from the container, run this command: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N emsvm update --offline

  • You may be prompted to reboot and run a second time if the kernel or OFED is changed.  

msg: 
 - 'Seems that kernel has changed. This will require a reboot ' 
 - Please exit container and reboot <MScontainer_hostname> 
 - Restart container (./essmkyml --restart) once ems is back and run update again.

  • Reboot the Utility node server with the following commands: 

ESS UNIFIED v6.2.x.x CONTAINER root@<MScontainer_hostname>:/ # exit 
[root@utilityBareMetal ~]# reboot

  • Once the server boots again validate that the VM is running: 

[root@utilityBareMetal ~]# virsh  list --state-running 
 Id   Name        State 
--------------------------- 
 1    EMSVM-23E   running

  • If the VM is not running, start it with the following commands: 

[root@utilityBareMetal ~]# cd /serv/EMSVM-main/ ; ./emsvm --start-EMS

  • Log into to the VM and restart the container: 

[root@emsvm ~]# cd /home/deploy/ess_6.2.x.x_...dir/ ; ./essmkyml --restart

  • Run update again: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ #essrun -N emsvm update --offline

  • You may encounter the following message: 

msg: 
  - Please shutdown <MScontainer_hostname> and reboot the Utility Host(Bare Metal) since OFED was updated or reinstalled 
  - "YOU MUST RUN THE FOLLOWING SCRIPT IN THE EMS VM AFTER REBOOT TO SET NETWORKING PARAMETERS AND ACTIVATE GPFS" 
  - "Run this script: '/opt/ibm/ess/tools/samples/reloadEms.sh' when <MScontainer_hostname> is back."

  • Reboot the node: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # ssh utilityBareMetal  
[root@utilityBareMetal ~]# reboot

  • When VM comes back run the script: 

[root@emsvm ~]# /opt/ibm/ess/tools/samples/reloadEms.sh

  • If a cluster exists in the environment, restart the performance monitoring and GUI: 

[root@emsvm ~]# systemctl restart pmsensors 
[root@emsvm ~]# systemctl restart gpfsgui

  1. Update I/O nodes after the EMSVM node is updated, from the container, run this command  

  • Restart container: 

[root@emsvm ~]# cd /home/deploy/ess_6.2.x.x_...dir/ ; ./essmkyml --restart

  • Update I/O nodes 

    • Offline mode: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N <IOnode1>,<IOnode2>,<IOnoden> update --offline

    • Online mode: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N <IOnode1>,<IOnode2>,<IOnoden> update

  • Restart the performance monitoring service on each upgrade I/O node: 

[root@IOnodeN ~]# systemctl restart pmsensors

  • Update enclosures firmware if it is needed 

    • Offline mode: 

[root@IOnodeN ~]# mmchfirmware --type storage-enclosure --fast-offline

    • Online mode: 

[root@IOnodeN ~]# mmchfirmware --type storage-enclosure

  • Update the drives firmware if it is needed  

    • Offline mode: 

[root@IOnodeN ~]# mmchfirmware –type drive --fast-offline

    • Online mode: 

[root@IOnodeN ~]# mmchfirmware --type drive

  • If the firmware needs to be updated in the I/O nodes run the following commands. NOTE: Just for POWER nodes 

[root@IOnodeN ~]# cd /install/ess/otherpkgs/rhels8/ppc64le/firmware/ 
[root@IOnodeN ~]# update_flash -v -f 01VL950_131_045.img 
[root@IOnodeN ~]# update_flash -f 01VL950_131_045.img 
[root@IOnodeN ~]# update_flash -c

  1. Update the Utility Protocol BareMetal server. NOTE: This step is only for VM protocol nodes 

  • Run from the container in EMSVM the following command: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N protocolBareMetal update --offline

  1. Update IBM Storage Scale in the protocol nodes.  

    • Offline mode. 

For more information about how to upgrade IBM Storage Scale in protocol nodes, see “Performing offline upgrade or excluding nodes from upgrade by using installation toolkit” 

    • Online mode. 

For more information about how to upgrade IBM Storage Scale in protocol nodes, see “Performing online upgrade by using the installation toolkit” 

  1. Upgrade protocol nodes operative system and other components using container  

    • Offline mode: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N prtvm1 update--offline

    • Online mode: 

ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # 
ESS UNIFIED v6.2.x.x CONTAINER root@containerHostname:/ # essrun -N prtvm1 update

  1. If the firmware needs to be updated in the protocol nodes run the following commands. NOTE: Just for POWER nodes 

[root@prtN ~]# cd /install/ess/otherpkgs/rhels8/ppc64le/firmware/ 
[root@prtN ~]# update_flash -v -f 01VL950_131_045.img 
[root@prtN ~]# update_flash -f 01VL950_131_045.img 
[root@prtN ~]# update_flash -c

  1. Run health checks on each node that were upgraded: 

essinstallcheck

Conclusion  

In conclusion, the update process described here shares many similarities with the approach detailed in our previous blog on POWER9 EMS. However, in this case, updating both the EMS VM and the BareMetal server introduces additional steps. 

We also explored the scenario of updating VM protocol nodes, which, like the EMS, requires attention to both the VM protocol and the BareMetal server to maintain system compatibility and functionality. 

For a comprehensive guide and further details on this procedure, be sure to consult the official documentation at: IBM Storage Scale System Deployment Guide 6.2.2 


#Highlights
#Highlights-home
0 comments
9 views

Permalink