Storage Fusion

Introduction

In this blog series we have focused on protecting applications with IBM Storage Fusion Backup & Restore and how to achieve application protection using recipes. But what about protecting the cluster itself? Specifically, can you protect the cluster by backing-up all of the cluster-scoped resources with Fusion Backup & Restore. The short answer is - yes, you can achieve a basic level of cluster resource protection with Fusion which can be useful for specific use cases.

IBM Storage Fusion Backup & Restore is designed to provide data protection for OpenShift applications and generally not the cluster itself. In other words, if you lose an application, you can recover it entirely using Fusion Backup & Restore. If you lose the entire cluster, you will still have to re-install the cluster without the aid of Fusion before you can start recovering the individual applications. 

Even though Fusion Backup & Restore does not help you recover the cluster itself, it can still be useful if you need to recover a specific set of cluster resources after the cluster has been re-installed. Let’s look at how to achieve this.

Backuping-up cluster-scoped resources

Backing-up cluster-scoped resources is fairly straight-forward. The recipe framework already provides a way to backup cluster-scoped resources by using the directive: includeClusterResources: true

Usually this directive is used with a subset of cluster-scoped resources, for example, a specific type of cluster-scoped resource and/or a set of cluster-scoped resources identified by a unique label.

It is possible to create a recipe to backup all of the cluster-scoped resources. This would be desirable as you will always have a backup of all the cluster-scoped resource in case you need to recover some of these resources outside of the scope of a normal application recovery. A simple, resource-only recipe can be used to achieve this

apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  name: cluster-scoped
spec:
  appType: fusion-backup-restore
  groups:
    - name: cluster_resources
      type: resource
      includeClusterResources: true
      restoreOverwriteResources: false
  workflows:
  - name: backup
    sequence:
    - group: cluster_resources
  - name: restore
    sequence:
    - group: cluster_resources

Note the inclusion of the includeClusterResources: true directive.

It is recommended that you create a new project to use to associate the cluster-scoped backup resources, for example ibm-fusion-cluster-protection.

After you create the project, you can assign a backup policy and associate the recipe above with the backup policy.

Validating the backup

To ensure you are capturing the cluster-scoped resources correctly, use the getResources tool to validate the resources that were processed in the backup. You should see thousands of resources without a namespace value in the script output which indicates a cluster-scoped resource, for example:

% getResources.sh backup -n ibm-fusion-cluster-protection-demo-q4h-apps.spparch.spp-ocp.tuc.stglabs.ibm.com-202404121511 | tail
,storageprofiles.cdi.kubevirt.io,ocs-storagecluster-cephfs,1
,storageprofiles.cdi.kubevirt.io,openshift-storage.noobaa.io,1
,storageprofiles.cdi.kubevirt.io,thin,1
,storageprofiles.cdi.kubevirt.io,thin-csi,1
,cdiconfigs.cdi.kubevirt.io,config,1
,storageversionmigrations.migration.k8s.io,flowcontrol-flowschema-storage-version-migration,1
,storageversionmigrations.migration.k8s.io,flowcontrol-prioritylevel-storage-version-migration,1
ibm-fusion-cluster-protection,recipes.spp-data-protection.isf.ibm.com,cluster-scoped,1
,helmchartrepositories.helm.openshift.io,openshift-helm-charts,1

Restoring cluster-scoped resources

Let’s assume you have the need to recover a specific cluster-scoped resource that is not associated with an application. How should you proceed?

It is not recommended that you recover all the cluster-scoped resources as you would an application. Even though the restore process will not overwrite existing resources, you might restore resources that although are no longer valid might have undesirable side affects (for example, an unwanted security exposure). Here are general guidelines you should follow if you want to recover cluster-scoped resources outside of an application recovery:

1. Don’t arbitrarily restore all of the cluster-scoped resources. This was noted above but worth repeating.

2. By default, ensure that existing cluster-scoped resources are not overwritten. You will notice the directive restoreOverwriteResources: false in the sample recipe provided above. Although this is not necessary (as the default value is false); it was provided in the recipe in case you need to change the value.

3. Only restore specific cluster-scoped resources by specifying the resource type and a unique label. While it is not always possible to identify the resource by a unique label, it is useful to restrict the restore request as much as possible.

4. Delete the cluster-scoped resource on the cluster if possible. Again, by default the restore operation will not overwrite an existing resource. By deleting the cluster-scoped resource first, you can still take advantage of not overwriting resources in case you can explicitly select the desired resource by a type (includeResourceTypes:) and label (labelSelector:). For example, if you want to restore a specific cluster-scoped resource of type clusteroperators.config.openshift.io and there is no label that allows you to isolate the resource, delete the resource on the cluster first and then restore all of the clusteroperators.config.openshift.io type resources without overwriting any existing resource.

5. Overwrite cluster-scoped resources only as a last resort. If you are not able to delete the resource from the cluster, you can specify a recovery recipe to use restoreOverwriteResources: true

Note that if you make changes to the recipe for recovery, you will need to specify the modified recipe in the restore resource when initiating the restore. For more information about specifying a unique recipe on restore, see the blog post Fusion Recipe Tips - I Didn’t Use a Recipe on Backup and My Application Won’t recover - What Can I Do?