Storage Fusion

This is a technical follow-up to the article OpenShift Virtualization - Backup and restore use cases with IBM Fusion FAQ. The intent of this article is to provide more detailed information regarding backup consistency with IBM Fusion 2.8.

Note: The IBM Fusion team is actively working on providing how-to documentation for backup and restore of virtual machines in the Knowledge Center.

Overview

When using IBM Fusion Backup and Restore services to protect virtual machines deployed through OpenShift Virtualization, there are a few options to provide data consistency within the virtual machine. There are three levels of data consistency that can be achieved in the virtual machine:

1. Crash consistency - this is achieved by taking a snapshot of the virtual machine PVCs without any additional orchestrations or hooks within the virtual machine.

2. File system consistency - this is achieved by tools inside the virtual machine interacting with the file system to freeze writes before a snapshot (backup) operation.

3. Application consistency - this is achieved by tools inside the virtual machine interacting with an application to take an application specific action such as locking a database.

In this article we will look start by looking at the default data consistency posture which is file system consistency. Let’s first look at the mechanics for backup operations with OpenShift Virtualization virtual machines using file system consistency.

File system consistency

In the diagram above, we have a virtual machine running within a namespace. Each virtual machine has a virt-launcher pod that runs the virtual machine instance. The virt-launcher pod runs an instance of libvirt, which is used to manage the virtual machine process.

When an IBM Fusion backup is requested for the namespace, a vm snapshot operation will be executed. The virt-launcher pod will have Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations to execute the kubevirt tool virt-freezer which communicates to a file system driver inside the Linux or Microsoft Windows virtual machine guest to freeze and unfreeze i/o operations around the virtual machine snapshot.

Configuring and validating file system consistency for Linux virtual machines

1. Ensure that virt-launcher pod has the appropriate Velero annotations:
   pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]'

   post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]

2. Ensure that the QEMU guest agent (qemu-guest-agent) is installed in the Linux virtual machine. Refer to OpenShift documentation.

3. Add the default service account security context constraint to “kubevirt-controller“:
   oc adm policy add-scc-to-user kubevirt-controller -z default -n <namespace>

4. Disable SELinux inside the virtual machine.

Configuring and validating file system consistency for Microsoft Windows virtual machines

1. Ensure that virt-launcher pod has the appropriate Velero annotations:
   pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]'

   post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]

2. Ensure that the QEMU guest agent (qemu-ga-x86_64) is installed in the Microsoft Windows virtual machine and that the “QEMU Guest Agent” network service is running. Refer to OpenShift documentation.

3. Ensure that the “Volume Shadow Copy” service is running in the Microsoft Windows virtual machine.

4. Add the default service account security context constraint to “kubevirt-controller“:
   oc adm policy add-scc-to-user kubevirt-controller -z default -n <namespace>

Crash consistency

Crash consistency can be achieved by taking a snapshot of the virtual machine without engaging the QEMU guest agent to perform a file system freeze. This can be done in one of two ways:

1. Remove the Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations from the virt-launcher pod. Note that this change will not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created when the virtual machine is created and will have to be removed for each new instantiation of the virt-launcher pod OR
   

2. Ensure that the QEMU guest agents are not installed and/or running in the virtual machine guest.

Application consistency

If you have specific application consistency requirements which cannot be met using the file system consistency mentioned above, you can create a backup workflow that communicates directly with the application using the virtual shell tool (virsh) instead of the virt-freezer tool:

Using Velero annotations to execute an application command using virsh

1. You can directly modify the Velero annotations on the virt-launcher pod to execute virsh (/usr/bin/virsh) instead of virt-freezer (/usr/bin/virt-freezer)

2. Note that modifying the Velero annotations will not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created when the virtual machine is created and will have to be modified for each new instantiation of the virt-launcher pod.

Using IBM Fusion backup and restore recipes to execute an application command using virsh

Alternatively, you can create an IBM Fusion backup and restore recipe to run an application command using virsh. There is no difference in the outcome as compared to running virsh through the Velero annotations but you might want to consider this approach and these points:

1. There might be instances that you have to execute more then one command to produce desired results, for example, issuing a command which locks a database for writes and a second command to monitor or validate the lock. In these cases a Fusion recipe might be easier to work with.

2. If you decide to use a Fusion recipe, the Velero annotations will also be honored which might produce undesirable results. For this reason, consider removing the Velero annotations from the virt-launcher pod
   

3. Use the recipe exec hooks to execute virsh in the desired sequence.

#Highlights

#Highlights-home