Install-toolkit can be used to upgrade core GPFS, NFS, SMB, CES, object, callhome, fileauditlogging, management GUI, and performance monitoring from the current version of IBM Spectrum Scale to the next higher version.
There was a limitation with install-toolkit prior to 5.0.2 release, where all nodes of the cluster must be active and all services on the nodes must be running. If any of the node is down or any of the service is not running, upgrade precheck detects the issue, and the toolkit doesn't allow you to proceed for the upgrade. In this situation, the offline upgrade feature is critical from a customer perspective, and that's why this feature is introduced in the 5.0.2 release. Now, Install toolkit is capable of doing upgrade (offline upgrade) in case of a node being unhealthy.
More details about marking a node as offline:
A node is considered to be unhealthy and can be added as offline, if node is reachable(pingable) but GPFS and all protocol components(SMB, OBJ, NFS) are down. However, if a node is not reachable, you cannot add this as offline, however, you may add this as excluded node.
Behaviour of install-toolkit on offline node:
The install-toolkit only upgrade the installed (gpfs,nfs,smb etc.) rpm's on that node. It does not attempt to start any services. i.e, mmstartup, mmces service start command is not invoked on that node.
Potential use cases for offline:
* Node is offline and is still needed to upgrade the node
* Offline nodes can reduce the upgrade duration (reduced checking)
* Keep the nodes offline to complete an OS upgrade
* Keep the nodes offline to complete other software updates (OFED, drivers etc)
* Keep the nodes offline so a reboot can be performed
* Offline all nodes to perform an offline upgrade
Following is the CLI command to add a node as offline:
root@vm1:/usr/lpp/mmfs/5.0.2.0/installer# ./spectrumscale upgrade config offline -h
Ex: ../spectrumscale upgrade config offline -N snafflefang-vm4.tuc.stglabs.ibm.com
Few validations are added for offline functionality:
root@snafflefang-vm1:/usr/lpp/mmfs/5.0.2.0/installer# ./spectrumscale upgrade config offline -N snafflefang-vm4.tuc.stglabs.ibm.com
[ FATAL ]You can't add this node as in offline because gpfs is active on this node.
root@snafflefang-vm1:/usr/lpp/mmfs/5.0.2.0/installer#./spectrumscale upgrade config offline -N snafflefang-vm4.tuc.stglabs.ibm.com
[ FATAL ] If snafflefang-vm4.tuc.stglabs.ibm.com is added in offline or excluded, quorum cannot be maintained in the cluster. Therefore, you cannot add this node in offline or excluded list.
Sample output layout of upgrade config list, after adding vm2 and vm4 as offline:
* You can add one or more(all) nodes of the cluster as offline.
* If you add a protocol node as offline, all services(SMB, NFS, OBJ) will be automatically added in offline. As you can see, offline keyword against each protocol service.
After adding a node as offline you can run upgrade precheck:
Once upgrade precheck is successful, you can run upgrade:
The upgrade process is divided into two phases. In Phase 1, all non-protocol nodes will be upgraded. In Phase 2, all protocol nodes will be upgraded. The upgrade process might cause a brief outage of Object, SMB, NFS, and Performance Monitoring components. Do you really want to begin upgrading? [y/N]:
…………….
…………….
[INFO] ---------------------------------------------------------
[INFO] Upgrade Success
[INFO] Successfully upgraded components: "license, gui, object, gpfs, nfs, ces, smb, zimon, callhome, fileauditlogging" on 4 node(s): "node-vm1.tuc.stglabs.ibm.com, node-vm2.tuc.stglabs.ibm.com, node-vm3.tuc.stglabs.ibm.com, node-vm4.tuc.stglabs.ibm.com" in 36 minute(s) and 22 second(s).
[INFO] ---------------------------------------------------------
[INFO] Post Upgrade recommendations
[INFO] ----------------------------
[WARN] The following nodes are still marked offline: snafflefang-vm2.tuc.stglabs.ibm.com ,node-vm4.tuc.stglabs.ibm.com. Run ./spectrumscale upgrade config clear to remove this node from the offline list once desired.
[WARN] The installation toolkit does not attempt to bring online any nodes that are marked offline. You can run the 'mmstartup' command to bring the nodes online and then run 'mmces service start' to start the desired service.
[ INFO ] ---------------------------------------------------------
[INFO] Once all nodes in the cluster have been successfully upgraded, consider running 'mmchconfig release=LATEST' to migrate the cluster configuration and enable new functionality throughout the cluster. Consider running 'mmchfs Device -V full' to enable all new functionality requiring different on-disk data structures. These commands commit the newly upgraded code levels to the cluster. Review the following topic in the Knowledge Center for further details 'Completing the upgrade to a new level of IBM Spectrum Scale'.
For more detail visit:
Performing offline upgrade or excluding nodes from upgrade using installation toolkit#StorageManagementandReporting#PrimaryStorage#Storage