Introduction

In previous “Fusion Recipe Tips” blogs, we have examined the use and various aspects of creating recipes to orchestrate backup and recovery workflows with IBM Storage Fusion. We have been assuming that you start to think about orchestration and recipe creation before you take the first backup of your application. But what if you have already taken backups and didn’t really think about orchestration? Can you still recover data and take advantage of the orchestration in the recipes?

To provide a little context - recall from previous articles that recipes aren’t always necessary to backup and recover your applications. Recipes are usually most useful for:

1. Applications that aren’t fully declarative (what we refer to as “imperative”). In these cases you might need to provide some level of orchestration to recover the application (either by eliminating certain resources from the restore and/or providing a sequencing of restore order for certain types of resources).
2. Providing data consistency via application consistency. In the case of active database applications, the backup recipe allows you to ensure that the data is application consistent before the backup snapshot operation by forcing a flush of data to disk or suspending write operations depending on the database engine.

In this article we are going to focus on the former: applications that aren't fully declarative. Let’s assume you have taken a backup of your application without a recipe. You might not have application consistency because no recipe was in place during the backup (but we will most likely have crash consistency which is sufficient for this discussion). Let’s further suppose you attempt a restore and the data is restored but the application doesn’t recover. We will look at how you can introduce a recipe into the restore workflow to orchestrate the recovery of resources so that your application can be properly recovered. 

Note that we will focus this article on the orchestration of resource recovery and assume that there is no additional attention that needs to be given to the PVCs. In other words, we are assuming you will need all the PVCs restored that you backed-up and not going to discuss how to further modify PVC recovery.

The Default Recipe

The first concept that is import to understand is that there is always a recipe associated with a Fusion backup operation, even if you did not specify a recipe in the policy assignment. This is known as the “default” recipe and there are two important keys about the default recipe that we need to understand:

1. The default recipe backs up all PVCs, namespace-scoped resources, and dependent cluster-scoped resources in the namespace. This is good news for recovery of the application because we have all the application data protected.

2. The default recipe uses the volume group name "vg” and a resource group name “rg” in the default recipe. This is important only because:

   •  the recipe processing will be expecting to find a volume group name “vg” if we create a restore recipe

   • we will need to refer to the group “rg” when we are trying to orchestrate resource recovery.

Example Recipe

Let’s assume you have an imperative application and that in order to recover the application you need to exclude some of the resources to ensure proper recovery. This is the recipe that should have been created before the first backup:

apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  name: db-backup-restore-recipe
  namespace: ibm-spectrum-fusion-ns
spec:
  appType: dbcluster
  groups:
    - name: db-volumes
      type: volume
      includedNamespaces:
        - mydb
    - name: db-resources
      type: resource
      includedNamespaces:
        - mydb
      excludedResourceTypes:
        - pods
        - replicasets
        - deployments
        - services
        - clusterserviceversions
  workflows:
  - name: backup
    sequence:
    - group: db-resources
    - group: edb-volumes
  - name: restore
    sequence:
    - group: db-volumes
    - group: db-resources

We said above that this should have been the recipe that you created for backup, but we are trying to rectify the case where we didn’t create the recipe. So our first step is to adapt the recipe above to the conventions used by the default recipe that was used for backup. This recipe will allow us to specify the intended resource orchestration on restore:

 

apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
kind: Recipe
metadata:
  name: db-backup-restore-recipe
  namespace: ibm-spectrum-fusion-ns
spec:
  appType: dbcluster
  groups:
    - name: vg
      type: volume
      includedNamespaces:
      - mydb
    - name: rg
      type: resource
      includedNamespaces:
      - mydb
    - name: db-resources
      type: resource
      backupRef: rg
      includedNamespaces:
      - mydb
      excludedResourceTypes:
      - pods
      - replicasets
      - deployments
      - services
      - clusterserviceversions
  workflows:
  - name: backup
    sequence:
    - group: vg
  - name: restore
    sequence:
    - group: vg
    - group: db-resources

A couple of important notes about the recipe above:

• We renamed the main volume group to "vg" to match the volume group name in the default recipe
• We renamed the main resource group to "rg" to match the resource group name in the default recipe
• We are using the resource group "db-resources" which allows us to provide orchestration during the restore. In this case we are going to only restore a subset of the resources (specifically all the resources except those that are listed in the excludedResourceTypes list). 
• Note that we have added a reference in the "db-resources" group to refer back to the original resource group "rg" using the backupRef filed.

Initiating the restore with the recipe

Now that we have the recipe, we have to create a restore request that uses the recipe.

First, apply the recipe you coded above with the oc apply -f <filename>.yaml

You might want to verify that the recipe was applied:

% oc apply -f fusion-blog-recipe.yaml
recipe.spp-data-protection.isf.ibm.com/db-backup-restore-recipe created

% oc get frcpe -n ibm-spectrum-fusion-ns
NAME                           AGE
db-backup-restore-recipe       20s

Next, we will need the backup name associated with the recovery point you want to recover. You can either do this by looking at the Fusion user interface and looi for the Name of the backup in the Details page associated with the desired recovery point or issue oc get fbackup -n ibm-spectrum-fusion-ns. The backup name usually takes the form of <application>-<policy>-<timestamp>, for example:

mydb-demo-3day-apps.spparch.spp-ocp.tuc.stglabs.ibm.com-202403121908

Finally, create a Fusion restore CR to initiate the restore request. You need to do this because the Fusion user interface currently does not allow you to initiate a restore request with a custom recipe name. You will need the recipe name and backup name from above to code the CR:

apiVersion: data-protection.isf.ibm.com/v1alpha1
kind: Restore
metadata:
  annotations:
    dp.isf.ibm.com/provider-name: isf-backup-restore
  name: mydb-recipe-restore-01
  namespace: ibm-spectrum-fusion-ns
spec:
  backup: mydb-demo-3day-apps.spparch.spp-ocp.tuc.stglabs.ibm.com-202403122023
  recipe:
    name: db-backup-restore-recipe
    namespace: ibm-spectrum-fusion-ns

After you submit this yaml file, you can track the restore progress in the Fusion user interface. Congratulations!  You have successfully orchestrated a restore of your application even though you did not use a recipe at backup time.