This is a technical follow-up to the article OpenShift Virtualization - Backup and restore use cases with IBM Fusion FAQ. The intent of this article is to provide more detailed information regarding backup consistency with IBM Fusion 2.8.

Note: The IBM Fusion team is actively working on providing how-to documentation for backup and restore of virtual machines in the Knowledge Center.

Overview

When using IBM Fusion Backup and Restore services to protect virtual machines deployed through OpenShift Virtualization, there are a few options to provide data consistency within the virtual machine. There are three levels of data consistency that can be achieved in the virtual machine:

1. Crash consistency - this is achieved by taking a snapshot of the virtual machine PVCs without any additional orchestrations or hooks within the virtual machine.

2. File system consistency - this is achieved by tools inside the virtual machine interacting with the file system to freeze writes before a snapshot (backup) operation.

3. Application consistency - this is achieved by tools inside the virtual machine interacting with an application to take an application specific action such as locking a database.

In this article we will look start by looking at the default data consistency posture which is file system consistency. Let’s first look at the mechanics for backup operations with OpenShift Virtualization virtual machines using file system consistency.

File system consistency

In the diagram above, we have a virtual machine running within a namespace. Each virtual machine has a virt-launcher pod that runs the virtual machine instance. The virt-launcher pod runs an instance of libvirt, which is used to manage the virtual machine process.

When an IBM Fusion backup is requested for the namespace, a vm snapshot operation will be executed. The virt-launcher pod will have Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations to execute the kubevirt tool virt-freezer which communicates to a file system driver inside the Linux or Microsoft Windows virtual machine guest to freeze and unfreeze i/o operations around the virtual machine snapshot.

Configuring and validating file system consistency for Linux virtual machines

1. Ensure that virt-launcher pod has the appropriate Velero annotations:
   pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]'

   post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]

2. Ensure that the QEMU guest agent (qemu-guest-agent) is installed in the Linux virtual machine. Refer to OpenShift documentation.

3. Add the default service account security context constraint to “kubevirt-controller“:
   oc adm policy add-scc-to-user kubevirt-controller -z default -n <namespace>

4. If the virtual machine operating system has SELinux in enforcing mode (or if you're planning to switch to enforcing mode), additional attention is needed to make sure that the qemu-guest-agent will always be able to freeze the all the file systems in the virtual machine:
   
   Check if the virtual machine guest operating system has SELinux in enforcing mode via:

   # sestatus

   Verify the status of the SELinux boolean virt_qemu_ga_read_nonsecurity_files via:

   # getsebool virt_qemu_ga_read_nonsecurity_files

   If this is set to off, enable it permanently via:

   # setsebool -P virt_qemu_ga_read_nonsecurity_files 1

   This will ensure that any additional disk or file systems that are mounted to the system, regardless of the SELinux label, will be able to support the freezed and unfreezed commands.

Configuring and validating file system consistency for Microsoft Windows virtual machines

1. Ensure that virt-launcher pod has the appropriate Velero annotations:
   pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]'

   post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--name", "rhel9-lovely-louse", "--namespace", "test-vm-ns"]

2. Ensure that the QEMU guest agent (qemu-ga-x86_64) is installed in the Microsoft Windows virtual machine and that the “QEMU Guest Agent” network service is running. Refer to OpenShift documentation.

3. Ensure that the “Volume Shadow Copy” service is running in the Microsoft Windows virtual machine.

4. Add the default service account security context constraint to “kubevirt-controller“:
   oc adm policy add-scc-to-user kubevirt-controller -z default -n <namespace>

Crash consistency

Crash consistency can be achieved by taking a snapshot of the virtual machine without engaging the QEMU guest agent to perform a file system freeze. This can be done in one of two ways:

1. Remove the Velero pre-snapshot (pre.hook.backup) and post-snapshot (post.hoook.backup) annotations from the virt-launcher pod. Note that this change will not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created when the virtual machine is created and will have to be removed for each new instantiation of the virt-launcher pod OR
   

2. Ensure that the QEMU guest agents are not installed and/or running in the virtual machine guest.

Application consistency

If you have specific application consistency requirements which cannot be met using the file system consistency mentioned above, you can create a backup workflow that communicates directly with the application using the virtual shell tool (virsh) instead of the virt-freezer tool:

Using Velero annotations to execute an application command using virsh

1. You can directly modify the Velero annotations on the virt-launcher pod to execute virsh (/usr/bin/virsh) instead of virt-freezer (/usr/bin/virt-freezer)

2. Note that modifying the Velero annotations will not persist across instantiations of the virt-launcher pod. The Velero annotations are automatically created when the virtual machine is created and will have to be modified for each new instantiation of the virt-launcher pod.

Using IBM Fusion backup and restore recipes to execute an application command using virsh

Alternatively, you can create an IBM Fusion backup and restore recipe to run an application command using virsh. There is no difference in the outcome as compared to running virsh through the Velero annotations but you might want to consider this approach and these points:

1. There might be instances that you have to execute more then one command to produce desired results, for example, issuing a command which locks a database for writes and a second command to monitor or validate the lock. In these cases a Fusion recipe might be easier to work with.

2. If you decide to use a Fusion recipe, the Velero annotations will also be honored which might produce undesirable results. For this reason, consider removing the Velero annotations from the virt-launcher pod
   

3. Use the recipe exec hooks to execute virsh in the desired sequence.

#Highlights

#Highlights-home