PowerVC - Remote Restart on PowerVM with NovaLink
PowerVM Remote VM Restart provides the ability to rebuild virtual machines (VMs) from a dead host (read: powered off) onto another (healthy) host within a single site. The combination of the PowerVC management capability and the PowerVM restart capability allows clients to very quickly restart workloads and provides additional options for availability and disaster recovery. The initial PowerVC support for remote VM restart was enabled on version 1.2.3 for HMC managed systems. With the introduction of PowerVC 1.3.1, this capability has been added for NovaLink managed hosts as well.
When PowerVC manages a PowerVM host through the HMC, it utilizes the native remote restart capability of PowerVM and the HMC. But with a NovaLink managed host, PowerVC instead leverages the native PowerVM OpenStack drivers ‘rebuild’ operation. This capability is similar to that of the HMC’s function, but has a couple of additional features. For example, for vSCSI connected volumes, it will ensure that the volumes are removed from the source host. NPIV based VMs also have an additional check to validate that the fabric shows the WWPNs offline before it allows a remote restart. From a user experience perspective, remote restart on NovaLink works in exactly the same way as in case of the HMC-managed scenario but it has a few more checks and protections built-in.
Rebuilding the VM
In order to properly rebuild any VM on a new host, several pieces of information are needed. The information needed to rebuild a VM are detailed below
- List of volumes assigned to a VM
- List of MAC addresses
- Storage zoning requirements
- Connectivity of storage to the new host
Fortunately for us, OpenStack natively stores most of this data.
However, PowerVM relies on two additional key pieces of information about a VM in order to precisely rebuild it – the NVRAM and the I/O Slot Configuration. The NVRAM is used to store important information, such as the boot list for the VM. The I/O Slot Configuration is used to indicate where adapters should be placed in the system.
The PowerVM OpenStack drivers have the capability to store all of the necessary configuration data to precisely rebuild the VM when paired with an object store called Swift. PowerVC makes use of this capability of the driver and packages the OpenStack Swift component with 1.3.1. All new installs of PowerVC as well as upgrades will gain this capability.
Let’s get into details of how to use the Remote Restart function from PowerVC.
Remote Restart Capabilities
Before we perform the remote restart operation we need to ensure that the hosts and VM being used for remote restart have the required capabilities enabled.
Host Capabilities
- The host must be enabled for simplified remote restart. By default, all NovaLink systems are enabled for simplified remote restart.
Virtual machine Capabilities
- In order to remote restart the VM, the Simplified remote restart capability must be enabled. All NovaLink VMs have the simplified remote restart flag turned on by default.
- To deploy a VM that does not have this capability on NovaLink use an advanced compute template with simplified remote restart capability turned off.
Remote Restart in Action
In a situation when one of the hosts (e.g. host B in figure below) being managed by PowerVC goes down, we may need to remote restart all important VMs on that host to another host. When the host is down or in error state, the “Remotely Restart Virtual Machines” option on the host list page gets enabled for the inoperative host. (Note this option is greyed out if the host is healthy)
To initiate a remote restart, select Host B and click Remotely Restart Virtual Machines. The Remote Restart Dialog appears with these options:
- Remotely restart all of the virtual machines on the host (also called “host-level remote restart”)
- Select an individual virtual machine to remotely restart.
img alt="" title="" lconnwikimacro="image" lconnwikiparamwikipage="Remote Restart on PowerVM with NovaLink" src="/powervc/wp-content/uploads/sites/110/2016/12/rrStep2.jpg" lconnwikiparamattachmentname="rrStep2.jpg">
Select VeryImportantVM to be remotely restarted. You are now given a choice of destination hosts. If you select “Selected by placement policy”, the scheduler will select one for you based on the placement policy in effect. The scheduler will only schedule restarts to hosts that have the same connectivity as the source.
Once the VM has been remote restarted, you will see a message and notification in the UI. Also, you can see that the host information is changed on the VM.
If you start a host-level remote restart but decide to cancel it, you can select the Remotely Restart Virtual Machines button again. It allows you to choose to cancel the currently running host-level remote restart or remote restart individual virtual machines while the host-level remote restart is in progress.
Remember, remote restart is needed only in emergency situations. That is, when a host has gone down. This can be jarring for your VM. For example, it may have failed in the middle of an I/O write operation and might need to be repaired. It is recommended that the admin review any VM that has gone down. This recovery would be similar to a power outage on a system so you can expect to have to fsck the file system and have the application go through recovery in the example of a database application.
Cleanup operation post remote restart
When the dead host is brought back up a cleanup operation is performed to delete the VM which has been successfully remote restarted to other host. This process is triggered as part of start of OpenStack Compute service on the NovaLink management partition and it cleans up the VM and any other resources that the VM may have been consuming.
Note - After bringing the host up it is recommended to put the host in maintenance mode and investigate the root cause for host crash or failure so that any further issues can be avoided.
We hope the information above helps in getting an understanding of the remote restart feature on PowerVM when managed with NovaLink! If you have any questions, feel free to post them below. And don’t forget to follow us on LinkedIn, Facebook, and Twitter!