PowerVC

PowerVC Rolling Upgrade

Introduction

In highly available clusters, one of the major challenges is upgrading existing software. While critical operations are in progress, the upgrading process might sometimes become a challenge considering downtime factors.

PowerVC 2.0.3 brings in the ‘rolling upgrade’ feature that helps you to upgrade from PowerVC 2.0.2 or 2.0.2.1. with negligible downtime or other disruption of services.

What is the significance of the rolling upgrade feature?

Now the HA is available in PowerVC 2.0.2 onwards, which intern enables rolling upgrade possibilities to keep the downtime at bay. In the PowerVC rolling upgrade approach, you upgrade one node at a time eventually resulting in the cloud running on the latest version. During this process, associated hosts and compute nodes are implicitly updated simultaneously when the PowerVC controller is updated.

What is this blog all about?

This blog covers details about the procedure for upgrading the OpsMgr utility and PowerVC on a single or multinode environment.

Any prereqs?

Yes. You must have PowerVC 2.0.2 or 2.0.2.1 installed before proceeding with the upgrade.

Help me understand more!
The image given below depicts the rolling upgrade process for single node and multinode.

Figure 1: Upgrade flow - Single node

Figure 2: Upgrade flow - Multinode

Before you upgrade...

Consider the following before upgrading PowerVC.

• PowerVC upgrade is always performed on PowerVC 2.0.2 or 2.0.2.1 installed nodes.
• A 3 node cluster upgrade happens node by node and has minimal downtime and limited operations are impacted during update•
• A node that is running ‘virtualip’ services is upgraded after the remaining nodes are upgraded.
• In a multinode environment, you must always run powervc-opsmgr update command on the primary node, and the upgrade is run on the node where OpsMgr is updated (primary node).
• Ensure that the core PowerVC services such as rabbitmq and MariaDB services are up and running before upgrading PowerVC.
• Update on a node can only be triggered after a successful update of one node. Under no circumstances should an update for another node be triggered if an update on one of the nodes fails.
• Clusters can still be used for basic operations such as deployment and basic life cycle operations while an upgrade is in progress.
• During the multinode upgrade process, you cannot add or remove new resources such as storage types, hosts and networks, etc., until all nodes are updated to version 2.0.3.
• VM clone, VM live capture, and clone volume are not supported till all nodes are upgraded to version 2.0.3.
• Any newly added APIs cannot be used till all 3 nodes are updated.
• If the update is interrupted at any point for various reasons, you can always retrigger the update procedure.
• If network disconnections are expected, try to upgrade by using nohup command.

Upgrading OpsMgr utility

Get the respective PowerVC OpsMgr archive and unarchive it.

After extracting the powervc-opsmgr-<OS-architecture-release>.tgz file, go to powervc-opsmgr-<version>/ directory.

Run update_opsmgr.sh script and accept the license. Alternatively, you can run <path>/update_opsmgr.sh -s to perform silent upgrade.

After successful execution of update_opsmgr.sh script, powervc-opsmgr rpm is upgraded to the latest on the primary node.

Upgrade PowerVC on a single node or on a multinode

After upgrading OpsMgr, you can upgrade PowerVC on a single node or on multiple nodes.

Run powervc-opsmgr update –help for more details on the update.

How to identify a VIP node?

In the case of a multinode upgrade, an update happens node by node with the VIP node being updated last.

You can use the pcs or the crm command and grep for ‘virtualip’ to know where ‘virtualip’ service is running. If you initiate an update on the VIP node at the beginning OpsMgr will report an error stating that the VIP is running on the node and recommends running an update on this node towards the end.

Upgrade a specific node

You can upgrade a specific node by running powervc-opsmgr update -c <cluster_name> -n <node_ip/hostname> .

Post PowerVC upgrade

• If monitoring is enabled before upgrading PowerVC, you must run powervc-opsmgr monitoring -c <cluster> --update to explicitly update monitoring; and thereby, update monitoring tools.
• Check if the UI shows the latest version of PowerVC.
• Check if Hosts, Storages, Networks, Virtual machines, etc. are in operating state and health is in 'OK' state.
• Monitor various service logs for any errors/failures after the update.
• File named powervc.lock might be generated at path /opt/ibm/powervc-opsmgr/powervc.lock to control operations. You can delete this file only when you are sure that no other operation is in progress and upgrade has failed without deleting the lock file (probably when the network disconnects).
• If any other operation has failed, the update operation will indicate the same and does not execute. The failed operation needs to be resolved before continuing update process.
• If any services/hosts are non-operational, running specific services restart is recommended.

Where to find PowerVC upgrade logs?

You can locate PowerVC upgrade logs at /opt/ibm/powervc-opsmgr/ansible/artifacts/