IBM Spectrum Fusion

 View Only

Upgrading IBM Spectrum Fusion 2.2.1 HCI to version 2.3.0

By ANSHU Garg posted Fri October 28, 2022 01:50 AM

  

Upgrading IBM Spectrum Fusion 2.2.1 HCI to version 2.3.0​


IBM Spectrum Fusion HCI (referred as ISF HCI here after) is Purpose-built, hyper-converged architecture designed to deploy bare metal Red Hat OpenShift container management and deployment software alongside IBM Spectrum Fusion software. For consistent and rapid deployment and management. For information on IBM Spectrum Fusion, visit https://www.ibm.com/docs/en/spectrum-fusion/2.3

ISF HCI 2.3.0 hit market on September 30th 2022.  Previous release was ISF HCI 2.2.1. In this article we walk you through the process of upgrading a previous version (2.2.1) of latest version (2.3.0). In Fusion HCI 2.3.0 upgrade is orchestrated by Fusion management software and provides seamless upgrade experience for it's services and firmware. 

In this article, we'll demonstrate how easy and intuitive it is to upgrade Fusion 2.2.1 to 2.3.0 from Fusion UI with it's events and banners that can be seen on different Fusion UI panels. 

Before you begin

Before you begin, it is strongly recommended to consult ISF HCI knowledge centre https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=fusion-upgrade-spectrum-hci-221-23 and complete all prerequisites before starting upgrade outlines in documentation.

IBM Spectrum Fusion Software 2.2.1 component versions

This section lists versions of various components in ISF for version 2.2.1 in table 1, which will be upgraded to versions shipped with 2.3.0.

Table 1

Component Version
IBM Spectrum Fusion management software 2.2.1
IBM Spectrum Scale (Global Data Platform) 5.1.3.1
IBM Spectrum Protect Plus (Data Protection) 10.1.10
Compute firmware 2.1.0
Switch firmware 4.3.0


Upgrade guide


Note: This article only covers upgrade in connected setup.

At high level, ISF HCI upgrade has following steps and in given order

  • ISF software upgrade
  • ISF services upgrade
    • Global Data Platform service (IBM Spectrum Scale) upgrade
    • Data Protection service (IBM Spectrum Protect Plus ) upgrade
  • Firmware upgrade
    • Compute firmware upgrade
    • Switch firmware upgrade
  • Red Hat OpenShift Container Platform upgrade to 4.9 
  • Red Hat OpenShift Container Platform (OCP) upgrade to 4.10

In next section we'll deep dive into each of the above steps.

ISF software upgrade


IBM Spectrum Fusion comes with it's management software that manages the HCI. This section describes steps required to initiate IBM Spectrum Fusion management software upgrade. For understanding more, refer to https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=usfh22-upgrading-spectrum-fusion-hci-management-software-from-221-23
​​​

Update catalog source to version 2.3.0

IBM Spectrum Fusion HCI comes with it's own catalog image. As first step for update catalogsource in OpenShift for HCI 2.2.1. Follow below steps to update catalogsource:

Before starting upgrade inspect installed IBM Spectrum Fusion version as seen in figure 1

Figure 1

List all catalogsources in OpenShift console of HCI 2.2.1 cluster as shown in figure 2.

 
Figure 2


Find isf-catalog in OpenShift console of HCI 2.2.1 cluster and edit image to ISF 2.3.0 image icr.io/cpopen/isf-operator-catalog:2.3.0-linux.amd64 as shown in figure 3 and save changes.


After couple minutes check installed IBM Spectrum Fusion operator again and now it will show that there is an upgrade available as can be seen in figure 4


Figure 4

Next, click on Upgrade available link and Approve Install Plan. It will take couple of minutes (~15 minutes) for upgrade to complete and will show two subscriptions in interim until 2.3.0 install completes as shown in figure 5.

Figure 5

Once operator upgrade is completed, it will be seen as Succeeded as can be seen in figure 6.
Figure 6

Now you are ready to proceed to services upgrade.

ISF service upgrade

IBM Spectrum Fusion comes with following two service. To learn more about these services, refer to documentation https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=managing-services

  • Global Data Platform
  • Data Protection

Detailed upgrade services steps are available at https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=23-upgrading-services


Global Data Platform service (IBM Spectrum Scale)upgrade


Global Data Platform refers to IBM Spectrum Fusion, which is the storage available with Fusion HCI.

Note: To upgrade Global data platform in Fusion 2.2.1 to Fusion version 2.3.0, you'll need to contact IBM Support.


Data Protection service (IBM Spectrum Protect Plus) upgrade


Data Platform refers to IBM Spectrum Protect Plus. To upgrade Data Protection service follow below steps.

First navigate to services page in Fusion UI as shown in figure 7.


Figure 7

Start Data Protection upgrade as shown in figure 8

Figure 8

It will take ~40 minutes to complete Data Protection upgrade.  Once, upgrade is completed, you can verify service status and updated version on Fusion UI service page as shown in figure 9.


Figure 9

Note: During upgrade, Data Protection will be offline.


Firmware upgrade


Fusion 2.3.0 ships newer versions of compute and switch firmware which include important updates and critical bug and security fixes. Read on to see how to upgrade these firmwares.

Compute firmware upgrade


Let's understand compute upgrade process briefly before jumping into upgrade steps. Fusion orchestrates the compute firmware upgrade such that user can carry out upgrades in batches of nodes. User can choose to schedule upgrade for select nodes at a time and once this pool is upgraded, can start next batch of upgrade. Within in scheduled pool, node compute firmware upgrade is sequential, that is, at a given point of time only one compute firmware is upgraded and rest are queued. 

IBM Spectrum Fusion knowledge centre has step by steps for compute firmware upgrade available at https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=nodes-upgrading-node-firmware.  We'll walk through same steps in this section with examples.

User can see notifications about compute firmware upgrade being available at top of Fusion UI at all times once upgrade becomes available after upgrade of Fusion management software is completed.  However, navigate to  Infrastructure-> Nodes page to view each node wise status and start upgrade as shown in figure 10.


Figure 10

To add multiple nodes to upgrade schedule, select nodes and click on Upgrade in Blue banner as seen in figure 11. You'll be presented with a popup for confirmation, click Start upgrade as can be seen in figure 11.


Figure 11

 You can see upgrade progress for each node and firmware level for all nodes from UI as shown in figure 12.


Figure 12

It takes approximately ~60 minutes for compute firmware upgrade to complete for each node

Following same steps, you can upgrade all nodes compute firmware easily from UI.

Switch firmware upgrade


Similar to compute firmware upgrade,  switch firmware upgrade is also orchestrated by Fusion management software. There are two types of switches used in Fusion. 

  • High speed
  • Management
Firmware upgrade is available only for high speed switches. Since Fusion comes with two switches for each type for High Availability and fault tolerance , upgrade is done in pair where secondary switch is upgraded first and then primary switch. Step by steps instructions are available at https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=overview-upgrading-switch-firmware for switch firmware upgrade.

As shown in figure 13, when switch firmware upgrade is available it can be seen from Fusion UI's Infrastructure-> Network page.


Figure 13

Click on Upgrade all button on Blue banner to start upgrade and you can monitor progress as shown in figure 14.


Figure 14

Note: It takes about two hours to complete switch firmware upgrade

This marks the completion of IBM Spectrum Fusion HCI 2.3.0 upgrade. Next comes Red Hat OpenShift Container Platform upgrade.

Red Hat OpenShift Container Platform 4.9 upgrade


Red Hat OpenShift can not be upgraded from 4.8 to 4.10 directly so we need to upgrade to 4.9 first and then to 4.10. Upgrade from 4.8 to 4.9 is straight forward and we suggest to follow Red Hat document for the same https://docs.openshift.com/container-platform/4.9/updating/index.html and complete upgrade to 4.9.

Red Hat OpenShift Container Platform 4.10 upgrade

Before starting upgrade to 4.10, there are couple steps that need to be taken since there have been changes between Red Hat OpenShift Container Platform 4.10 and earlier version of Red Hat OpenShift Container Platform network configuration with respect to interface br-ex, ovs and bonds. We strongly encourage reading more about these changes here respectively https://access.redhat.com/solutions/6972925 and https://docs.openshift.com/container-platform/4.10/installing/installing_bare_metal_ipi/ipi-install-post-installation-configuration.html#manual-migration-to-new-customdeploy-install-method_ipi-install-post-installation-configuration


Migrate to new OVS configuration for OCP 4.10


As mentioned above, starting with Red Hat OpenShift Container Platform 4.10, on node reboot, the ovs-configuration.service script tears down br-ex and reconfigures the main bond interface that was attached to br-ex. At that point in time, the bond's second slave member might be enslaved first and the recreated bond interface will use the secondary's MAC address. Subsequent attempts to obtain a DHCP lease - and thus a default route - might fail. To avoid that some additional configuration on cluster is required before OCP 4.10 upgrade is started. Follow Red Hat solution https://access.redhat.com/solutions/6972925

Note: Do not proceed to next step without verifying successful completion of this step

Remove old bond configuration


Yes another change in Red Hat OpenShift Container Platform 4.10 causes it to try to configure bonds on non existent interfaces since interfaces files are present on each CoreOS node. This causes network configuration failure during reboot for OCP upgrade to 4.10. To overcome this, it is required to delete those file from each node. Follow step 6.b from https://www.ibm.com/docs/en/spectrum-fusion/2.3?topic=hci-upgrade-spectrum-fusion-221-23

Now you are ready to upgrade Fusion HCI cluster to 4.10.

Upgrade OCP to 4.10


Follow Red Hat OpenShift document to upgrade OCP to 4.10. https://docs.openshift.com/container-platform/4.10/updating/updating-cluster-within-minor.html#update-upgrading-web_updating-cluster-within-minor

Note: The Red Hat OpenShift Container Platform version must be upgraded only to the version supported by IBM Spectrum Fusion HCI. At time of writing this article, we upgraded cluster to 4.10.32. 

Update machine set (for future upsize)


Starting Red Hat OpenShift 4.10, network customisation mechanism has changed, therefore, after upgrading Fusion cluster to 4.10, it is required it is migrated to new customDeploy install method. This will get info effect when new nodes are added to cluster after upgrade. Read more about migration here https://docs.openshift.com/container-platform/4.10/installing/installing_bare_metal_ipi/ipi-install-post-installation-configuration.html#manual-migration-to-new-customdeploy-install-method_ipi-install-post-installation-configuration

For this migration, login to Fusion OCP cluster using OpenShift client (oc)

Get machineset oc get machinesets -n openshift-machine-api

Edit machine set oc edit machineset <machineset> -n openshift-machine-api

Change spec to include below and save. Change is setting checksum and url to empty string. Leave rest of spec unchanged.

spec:
  providerSpec:
    value:
      customDeploy:
        method: install_coreos
      image:
        checksum: ""
        url: ""


Once change is applied, you can add new nodes to cluster.

Note: Do not add nodes to cluster before completing all of OCP 4.10 upgrade steps.

Monitor upgrade

Live progress of each component service and firmware upgrade can be monitored from Fusion UI respective page. It updates upgrade progress in % periodically for all upgrades as shown in figure 15 and 16.


Figure 15


Figure 16

Events are also generated for significant events for example when upgrade
  • Becomes available
  • Has been initiated
  • Completes
  • Fails

Verify upgrade

Once upgrade is completed you should be able to see following versions for each of components in IBM Spectrum Fusion 2.3.0

Component Version
IBM Spectrum Fusion management software 2.3.0
IBM Spectrum Scale (Global Data Platform) 5.1.5
IBM Spectrum Protect Plus (Data Protection) 10.1.12
Compute firmware 2.3.0
Switch firmware4.3.0 5.1.0

This concludes IBM Spectrum Fusion HCI 2.2.1 to 2.3.0 upgrade. As shown via this article fusion upgrade has been orchestrated to simplify upgrade experience. ​​​​​​​​
0 comments
31 views

Permalink