Upgrading IBM Spectrum Fusion 2.2.1 HCI to version 2.3.0
IBM Spectrum Fusion HCI (referred as ISF HCI here after) is
Purpose-built, hyper-converged architecture designed to deploy bare metal Red Hat OpenShift container management and deployment software alongside IBM Spectrum Fusion software. For consistent and rapid deployment and management. For information on IBM Spectrum Fusion, visit
https://www.ibm.com/docs/en/spectrum-fusion/2.3ISF HCI 2.3.0 hit market on September 30th 2022. Previous release was ISF HCI 2.2.1. In this article we walk you through the process of upgrading a previous version (2.2.1) of latest version (2.3.0). In Fusion HCI 2.3.0 upgrade is orchestrated by Fusion management software and provides seamless upgrade experience for it's services and firmware.
In this article, we'll demonstrate how easy and intuitive it is to upgrade Fusion 2.2.1 to 2.3.0 from Fusion UI with it's events and banners that can be seen on different Fusion UI panels.
Before you begin
Before you begin, it is strongly recommended to consult ISF HCI knowledge centre
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=fusion-upgrade-spectrum-hci-221-23 and complete all prerequisites before starting upgrade outlines in documentation.
IBM Spectrum Fusion Software 2.2.1 component versions
This section lists versions of various components in ISF for version 2.2.1 in table 1, which will be upgraded to versions shipped with 2.3.0.
Table 1
| Component |
Version |
| IBM Spectrum Fusion management software |
2.2.1 |
| IBM Spectrum Scale (Global Data Platform) |
5.1.3.1 |
| IBM Spectrum Protect Plus (Data Protection) |
10.1.10 |
| Compute firmware |
2.1.0 |
| Switch firmware |
4.3.0 |
Upgrade guide
Note: This article only covers upgrade in connected setup.
At high level, ISF HCI upgrade has following steps and in given order
- ISF software upgrade
- ISF services upgrade
- Global Data Platform service (IBM Spectrum Scale) upgrade
- Data Protection service (IBM Spectrum Protect Plus ) upgrade
- Firmware upgrade
- Compute firmware upgrade
- Switch firmware upgrade
- Red Hat OpenShift Container Platform upgrade to 4.9
- Red Hat OpenShift Container Platform (OCP) upgrade to 4.10
In next section we'll deep dive into each of the above steps.
ISF software upgrade
IBM Spectrum Fusion comes with it's management software that manages the HCI. This section describes steps required to initiate IBM Spectrum Fusion management software upgrade. For understanding more, refer to
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=usfh22-upgrading-spectrum-fusion-hci-management-software-from-221-23
Update catalog source to version 2.3.0
IBM Spectrum Fusion HCI comes with it's own catalog image. As first step for update catalogsource in OpenShift for HCI 2.2.1. Follow below steps to update catalogsource:
Before starting upgrade inspect installed IBM Spectrum Fusion version as seen in figure 1
Figure 1
List all catalogsources in OpenShift console of HCI 2.2.1 cluster as shown in figure 2.
Figure 2
Find isf-catalog in OpenShift console of HCI 2.2.1 cluster and edit image to ISF 2.3.0 image
icr.io/cpopen/isf-operator-catalog:2.3.0-linux.amd64 as shown in figure 3 and save changes.
After couple minutes check installed IBM Spectrum Fusion operator again and now it will show that there is an
upgrade available as can be seen in figure 4
Figure 4
Next, click on
Upgrade available link and Approve
Install Plan. It will take couple of minutes (~15 minutes) for upgrade to complete and will show two subscriptions in interim until 2.3.0 install completes as shown in figure 5.
Figure 5
Once operator upgrade is completed, it will be seen as Succeeded as can be seen in figure 6.
Figure 6
Now you are ready to proceed to services upgrade.
ISF service upgrade
IBM Spectrum Fusion comes with following two service. To learn more about these services, refer to documentation https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=managing-services
- Global Data Platform
- Data Protection
Detailed upgrade services steps are available at
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=23-upgrading-services.
Global Data Platform service (IBM Spectrum Scale)upgrade
Global Data Platform refers to IBM Spectrum Fusion, which is the storage available with Fusion HCI.
Note: To upgrade Global data platform in Fusion 2.2.1 to Fusion version 2.3.0, you'll need to contact IBM Support.
Data Protection service (IBM Spectrum Protect Plus) upgrade
Data Platform refers to IBM Spectrum Protect Plus. To upgrade Data Protection service follow below steps.
First navigate to services page in Fusion UI as shown in figure 7.
Figure 7
Start Data Protection upgrade as shown in figure 8
Figure 8
It will take ~40 minutes to complete Data Protection upgrade. Once, upgrade is completed, you can verify service status and updated version on Fusion UI service page as shown in figure 9.
Figure 9
Note: During upgrade, Data Protection will be offline.
Firmware upgrade
Fusion 2.3.0 ships newer versions of compute and switch firmware which include important updates and critical bug and security fixes. Read on to see how to upgrade these firmwares.
Compute firmware upgrade
Let's understand compute upgrade process briefly before jumping into upgrade steps. Fusion orchestrates the compute firmware upgrade such that user can carry out upgrades in batches of nodes. User can choose to schedule upgrade for select nodes at a time and once this pool is upgraded, can start next batch of upgrade. Within in scheduled pool, node compute firmware upgrade is sequential, that is, at a given point of time only one compute firmware is upgraded and rest are queued.
IBM Spectrum Fusion knowledge centre has step by steps for compute firmware upgrade available at
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=nodes-upgrading-node-firmware. We'll walk through same steps in this section with examples.
User can see notifications about compute firmware upgrade being available at top of Fusion UI at all times once upgrade becomes available after upgrade of Fusion management software is completed. However, navigate to
Infrastructure->
Nodes page to view each node wise status and start upgrade as shown in figure 10.
Figure 10
To add multiple nodes to upgrade schedule,
select nodes and click on
Upgrade in Blue banner as seen in figure 11. You'll be presented with a popup for confirmation, click
Start upgrade as can be seen in figure 11.
Figure 11
You can see upgrade progress for each node and firmware level for all nodes from UI as shown in figure 12.
Figure 12
It takes approximately ~60 minutes for compute firmware upgrade to complete for each node
Following same steps, you can upgrade all nodes compute firmware easily from UI.
Switch firmware upgrade
Similar to compute firmware upgrade, switch firmware upgrade is also orchestrated by Fusion management software. There are two types of switches used in Fusion.
Firmware upgrade is available only for high speed switches. Since Fusion comes with two switches for each type for High Availability and fault tolerance , upgrade is done in pair where secondary switch is upgraded first and then primary switch. Step by steps instructions are available at
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=overview-upgrading-switch-firmware for switch firmware upgrade.
As shown in figure 13, when switch firmware upgrade is available it can be seen from Fusion UI's
Infrastructure->
Network page.
Figure 13
Click on
Upgrade all button on Blue banner to start upgrade and you can monitor progress as shown in figure 14.
Figure 14
Note: It takes about two hours to complete switch firmware upgrade
This marks the completion of IBM Spectrum Fusion HCI 2.3.0 upgrade. Next comes Red Hat OpenShift Container Platform upgrade.
Red Hat OpenShift Container Platform 4.9 upgrade
Red Hat OpenShift can not be upgraded from 4.8 to 4.10 directly so we need to upgrade to 4.9 first and then to 4.10. Upgrade from 4.8 to 4.9 is straight forward and we suggest to follow Red Hat document for the same
https://docs.openshift.com/container-platform/4.9/updating/index.html and complete upgrade to 4.9.
Red Hat OpenShift Container Platform 4.10 upgrade
Before starting upgrade to 4.10, there are couple steps that need to be taken since there have been changes between Red Hat OpenShift Container Platform 4.10 and earlier version of Red Hat OpenShift Container Platform network configuration with respect to interface br-ex, ovs and bonds. We strongly encourage reading more about these changes here respectively
https://access.redhat.com/solutions/6972925 and https://docs.openshift.com/container-platform/4.10/installing/installing_bare_metal_ipi/ipi-install-post-installation-configuration.html#manual-migration-to-new-customdeploy-install-method_ipi-install-post-installation-configuration
Migrate to new OVS configuration for OCP 4.10
As mentioned above, starting with Red Hat OpenShift Container Platform 4.10, on node reboot, the
ovs-configuration.service script tears down
br-ex and reconfigures the main bond interface that was attached to
br-ex. At that point in time, the bond's second slave member might be enslaved first and the recreated bond interface will use the secondary's MAC address. Subsequent attempts to obtain a DHCP lease - and thus a default route - might fail. To avoid that some additional configuration on cluster is required before OCP 4.10 upgrade is started. Follow Red Hat solution
https://access.redhat.com/solutions/6972925Note: Do not proceed to next step without verifying successful completion of this step
Remove old bond configuration
Yes another change in Red Hat OpenShift Container Platform 4.10 causes it to try to configure bonds on non existent interfaces since interfaces files are present on each CoreOS node. This causes network configuration failure during reboot for OCP upgrade to 4.10. To overcome this, it is required to delete those file from each node. Follow step 6.b from
https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=hci-upgrade-spectrum-fusion-221-23Now you are ready to upgrade Fusion HCI cluster to 4.10.
Upgrade OCP to 4.10
Follow Red Hat OpenShift document to upgrade OCP to 4.10.
https://docs.openshift.com/container-platform/4.10/updating/updating-cluster-within-minor.html#update-upgrading-web_updating-cluster-within-minorNote: The Red Hat OpenShift Container Platform version must be upgraded only to the version supported by IBM Spectrum Fusion HCI. At time of writing this article, we upgraded cluster to 4.10.32.
Update machine set (for future upsize)
Starting Red Hat OpenShift 4.10, network customisation mechanism has changed, therefore, after upgrading Fusion cluster to 4.10, it is required it is migrated to new customDeploy install method. This will get info effect when new nodes are added to cluster after upgrade. Read more about migration here
https://docs.openshift.com/container-platform/4.10/installing/installing_bare_metal_ipi/ipi-install-post-installation-configuration.html#manual-migration-to-new-customdeploy-install-method_ipi-install-post-installation-configurationFor this migration, login to Fusion OCP cluster using OpenShift client (oc)
Get machineset
oc get machinesets -n openshift-machine-apiEdit machine set
oc edit machineset <machineset> -n openshift-machine-apiChange spec to include below and save. Change is setting
checksum and
url to empty string. Leave rest of spec unchanged.
spec: providerSpec: value: customDeploy: method: install_coreos image: checksum: "" url: ""Once change is applied, you can add new nodes to cluster.
Note: Do not add nodes to cluster before completing all of OCP 4.10 upgrade steps.
Monitor upgrade
Live progress of each component service and firmware upgrade can be monitored from Fusion UI respective page. It updates upgrade progress in % periodically for all upgrades as shown in figure 15 and 16.
Figure 15
Figure 16
Events are also generated for significant events for example when upgrade
- Becomes available
- Has been initiated
- Completes
- Fails
Verify upgrade
Once upgrade is completed you should be able to see following versions for each of components in IBM Spectrum Fusion 2.3.0
| Component |
Version |
| IBM Spectrum Fusion management software |
2.3.0 |
| IBM Spectrum Scale (Global Data Platform) |
5.1.5 |
| IBM Spectrum Protect Plus (Data Protection) |
10.1.12 |
| Compute firmware |
2.3.0 |
| Switch firmware4.3.0 |
5.1.0 |
This concludes IBM Spectrum Fusion HCI 2.2.1 to 2.3.0 upgrade. As shown via this article fusion upgrade has been orchestrated to simplify upgrade experience.