1. Introduction

To protect the environment from data loss and corruption, it is important to backup the environment, so that the environment could be rebuilt from the backup data if the primary environment failed to provide services when disaster happened.

This document describes sample steps to backup and restore Business Automation Workflow in IBM Cloud Pak for Business Automation 21.0.3 environment to a different openshift environment. More information could reference the IBM Knowledge center.

2. General idea

For BAW (Business Automation Workflow), please consider to backup and restore following data:

• Necessary PVC definition
• Necessary Contents on PV
• Necessary Secrets
• Databases
• CR files

It won’t backup and restore following things, they need to be installed manually on the restore site:

• Openshit platform
• IAF and IBM common services

 The deployment may be different in different user’s environment, so the backup and restore method for that environment maybe some different from others. In below section, it is using following configuration and method to backup and restore:

• Use Offline backup method. i.e. stop related pods before backup, so that there is no new data would be generated during backup, and the backup data could be in consistency state.
• The deployment is using NFS as storage class, so the backup and restore method here based on the file system.
• The deployment is using dynamic provision.
• The deployment is BAW enterprise pattern for CP4BA 21.0.3 with Business Automation Workflow Capability and Content Platform Engine Capability.
• The deployment is using DB2 single server as database server, and there is not configured JDBC over SSL.

3. Backup on the primary environment

Assumed the BAW enterprise pattern has been setup and configured correctly on primary site. Please reference below steps to backup the primary environment.

1. The Cloud Pak custom resource (CR) file used to deploy the environment.
2. The secret definition that associated with the CR file. For example, the database username and password etc.
3. Backup the uid definition for the primary environment. For example, execute below commands and the output may be a string like 1000670000.

   oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1
   
   Note: $NS is the namespace that deployed the BAW environment.

4. Retrieve the value for WLP_CLIENT_ID and WLP_CLIENT_SECRET from secret ibm-common-services/platform-oidc-credentials. It could be retrieved from openshift console or retrieved with following commands for example.
   

   oc -n ibm-common-services get secret platform-oidc-credentials -o jsonpath='{.data.WLP_CLIENT_ID}' | base64 -d 
   oc -n ibm-common-services get secret platform-oidc-credentials -o jsonpath='{.data.WLP_CLIENT_SECRET}' | base64 -d
   

5. Backup the secret admin-user-details. For example, using below commands to backup.

   kubectl get secret admin-user-details -o yaml \
   | yq eval 'del(.metadata.annotations, .metadata.creationTimestamp, .metadata.ownerReferences, .metadata.resourceVersion, .metadata.uid)' - > admin-user-details.yaml
   

6. Stop the environment before backup runtime data. See below commands for example, please note, the “demo” in commands is the ICP4ACluster name that defined in the CR file.

   oc scale deploy ibm-cp4a-operator --replicas=0
   for i in `oc get deploy -o name |grep demo`; oc scale $i --replicas=0; done
   for i in `oc get sts -o name |grep demo`; oc scale $i --replicas=0; done
   oc scale sts zen-metastoredb --replicas=0 
   

7. Backup PVC definition for following PVC:

   • cpe-filestore-pvc-ctnrs
   • icn-cfgstore-ctnrs
   • datadir-zen-metastoredb-0
   • datadir-zen-metastoredb-1
   • datadir-zen-metastoredb-2
   • demo-instance1-baw-jms-data-vc-demo-instance1-baw-jms-0
   For static PV, you owned every PVC definition, so you can keep a copy of it directly. For dynamic provision, PVC/PV definition would be created by operator automatically, below commands give an example on how to retrieve the pvc definition for above PVC:

   pvc_file="pvc.yaml"
   
   oc get pvc -n $NS --no-headers=true | while read each
   do
       pvc=`echo $each | awk '{ print $1 }'`
       pvc_backup=false
   
       case $pvc in
           *"cpe-filestore"*)  pvc_backup=true ;;
           *"datadir-zen-metastoredb-"*)  pvc_backup=true ;;
           *"icn-cfgstore"*)  pvc_backup=true ;;
           *"baw-jms-data-"*)  pvc_backup=true ;;
           *"jms-pvc-"*)  pvc_backup=true ;;
           *) continue ;;
       esac
   
       if [ "$pvc_backup" = true ] ; then
           echo "  backup pvc yaml for $pvc"
           kubectl get pvc $pvc -o yaml \
             | yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' -  >> $BACKUP_DATA/$pvc_file
           echo "---" >> $BACKUP_DATA/$pvc_file
       fi
   done
   

8. Backup content for PV.

   For dynamic PV, it is not a static name for the generated directory, for example, the directory name may be “bawent-cmis-cfgstore-pvc-ctnrs-pvc-e5241e0c-3811-4c0d-8d0f-cb66dd67f672”, the directory name would be different for different deployment. Here is to use a “mapping” directory to backup the content.

   You could retrieve and backup each PV directory manually. Please note, it’s better to preserve permissions for all content during backup.

   Below is a sample script to create a backup. It would backup content for all PVC/PV, and it would restore the necessary PVs only during restore procedure.

   oc get pvc -n $NS --no-headers=true | while read each
   
   do
       pvc=`echo $each | awk '{ print $1 }'`
       pv=`echo $each | awk '{ print $3 }'`
   
       if [  -d "$SOURCE_PV_DIR/$NS-$pvc-$pv" ]
       then
           echo "  backup pv $pv for pvc $pvc"
           mkdir -p “$BACKUP_DIR/pvfiles/$pvc”
           cp  -r  -a $SOURCE_PV_DIR/$NS-$pvc-$pv/.  “$BACKUP_DIR/pvfiles/$pvc”
       else
           echo "  NOT FOUND for $pvc !"
       fi
   done
   
   cd
   echo -e "\n  compress the PV backup files  into $BACKUP_DIR/pv.tgz ..."
   cd $BACKUP_DIR
   tar cpzf /$BACKUP_DIR/pv.tgz  ./pvfiles/*

   Please note, in above scripts, for the variable:

   • $NS: it is the namespace that deployed the BAW environment.
   • $SOURCE_PV_DIR: the location where the NFS mounted directory.
   • $BACKUP_DIR: the location used to save the backup files and data.

9. Backup BTS. Assumed that the name of BTS pod is ibm-bts-cnpg-bawent-cp4ba-bts-1, execute following commands, please copy bk_btsdb.sql outside of the pod and backup it.
   Please note, BTS could backup data into S3 compatible storage, here is using another method to backup and restore BTS data.

   oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1
   export PX_USER=$(cat /etc/superuser-secret/username)
   export PX_PASSWORD=$(cat /etc/superuser-secret/password)
   pg_dump -d BTSDB -U postgres -Fp  -c -C --if-exists  -f /var/lib/postgresql/data/bk_btsdb.sql  

10. Backup Databases, this could be done with database commands.
11. Copy all above backup files and data to the remote site.

Prerequisite before restore:

1. IAF/common services should be installed.
2. Create a openshift project with exactly the same name as the primary environment.

 Please reference below steps to restore the environment.

1. Restore all the databases from the backup images.
2. Restore PVC, apply the generated pvc.yaml.
3. Restore content of PV.

   1. Extract the pv.tgz with preserved file permissions.

      tar  cpzf  pv.tgz  ./pvfiles/*

   2. Check the uid used for the primary environment and restored environment. For example, execute below commands on both environments.

      oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1

      The output may be :

      For primary environment, the uid is 1000670000

      For restore environment, the uid is 1000640000

   3. For the exacted files, modify file permissions to associate with restored environment. E.g. , modify uid to 100064000 for the files whose uid is 100067000:
      

      find ./  -uid  1000670000  -exec chown 1000640000:root {} \;

   4. Restore the PV content for following PVC:

      • cpe-filestore-pvc-ctnrs
      • icn-cfgstore-ctnrs
      • datadir-zen-metastoredb-0
      • datadir-zen-metastoredb-1
      • datadir-zen-metastoredb-2
      • demo-instance1-baw-jms-data-vc-demo-instance1-baw-jms-0

   5. Copy all corresponding files for above PVs with same permission. For example, Below is a sample script to restore content for the necessary PV.
      

      NS=bawent
      BACKUP_DIR=/home/backup
      BACKUP_PV_DIR=$BACKUP_DIR/pvfiles
      TARGET_PV_DIR=/home/pv/2103
      
      if [ ! -d $BACKUP_PV_DIR ] ; then
          echo "Not found backup files! "
          exit
      fi
       
      oc get pvc -n $NS --no-headers=true | while read each
      do
          pvc=`echo $each | awk '{ print $1 }'`
          pv=`echo $each | awk '{ print $3 }'`
          # echo "$pvc : $pv "
      
          case $pvc in
              *"cpe-filestore"*)  echo "restore $pvc" ;;
              *"datadir-zen-metastoredb-"*)  echo "restore $pvc" ;;
              *"icn-cfgstore"*)  echo "restore $pvc" ;;
              *"baw-jms-data-"*)  echo "restore $pvc" ;;
              *"jms-pvc-"*)  echo "restore $pvc" ;;
              *) continue ;;
          esac
      
          if [  -d "$BACKUP_PV_DIR/$pvc" ]
          then
              echo "  cp -r -a $BACKUP_PV_DIR/$pvc/.  $TARGET_PV_DIR/$NS-$pvc-$pv/ "
              cp -r -a $BACKUP_PV_DIR/$pvc/*  $TARGET_PV_DIR/$NS-$pvc-$pv/
          else
              echo "NOT FOUND for $pvc"
          fi
      done

4. Restore secrets. Create the secure from the backup secure definition, including the secret admin-user-details.

5. Prepare the CR definition for the restored environment.

   1. The hostname might be different between the source environment and restored environment. If they are different, you can change the hostname in the CR files to match the hostname of the restored environment.
   2. If initialization has been done in original deployment, set “shared_configuration.sc_content_initialization: false” and “shared_configuration.sc_content_verification: false”to avoid redo initialization and verification to break existing data in database.
   3. If BAI is enabled, please remove initialize_configuration section in CR.

    

6. Apply the CR to deploy the environment

7. Restore IAM registration. This could be done during the deployment.

Note: Below steps need to be run after IAM is ready, IAM status can be observed by the "oc describe configmap ibm-common-services-status -n kube-public", if it shows "iamstatus: Ready", then it is ready to do below steps. This step was improved in the next release.

1. 1. find the ibm-common-services/registration-json configmap on the restored cluster and save it as platform-oidc-registration.json file.
   2. update the client_id and client_secret in platform-oidc-registration.json file with the value of WLP_CLIENT_ID and WLP_CLIENT_SECRET we got from above backup step.
   3. Follow https://www.ibm.com/docs/en/cpfs?topic=apis-oidc-registration#register to register with platform-oidc-registration.json generated in step 2.

Below script is an example to do the registration by commands, execute following commands during CR deployment.

1. echo -e "\nWaiting configmap registration-json to be created ... "
   while true
   do
       if oc -n ibm-common-services get configmap | grep -q registration-json ; then
           echo "  configmap registration-json was created! waiting 180 sec ..."
           sleep 180
           break
       else
           sleep 15
       fi
   done
    
   echo -e "\nGet & update platform-oidc-registration.json"
   oc -n ibm-common-services get configmap registration-json  -o jsonpath='{.data.*}'  >   $BACKUP_DATA/platform-oidc-registration.json
    
   WID=`cat $BACKUP_DATA/src_WLP_CLIENT_ID`
   jq  '."client_id" = "'"$WID"'"'  $BACKUP_DATA/platform-oidc-registration.json > tmp  &&  mv tmp $BACKUP_DATA/platform-oidc-registration.json
    
   WST=`cat $BACKUP_DATA/src_WLP_CLIENT_SECRET`
   jq  '."client_secret" = "'"$WST"'"'  $BACKUP_DATA/platform-oidc-registration.json > tmp  &&  mv tmp $BACKUP_DATA/platform-oidc-registration.json
    
    
   echo -e "\nGet access_token ..."
   iam_user=`oc -n ibm-common-services get secret ibm-iam-bindinfo-platform-auth-idp-credentials  -o jsonpath='{.data.admin_username}' | base64 -d  `
    
   iam_pwd=`oc -n ibm-common-services get secret ibm-iam-bindinfo-platform-auth-idp-credentials  -o jsonpath='{.data.admin_password}' | base64 -d`
    
   iam_cluster_address=`oc get route -n ibm-common-services cp-console -o jsonpath={.spec.host}`
    
   iam_json_data=`curl -k -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' \
     -d "grant_type=password&username=$iam_user&password=$iam_pwd&scope=openid" \
     https://$iam_cluster_address/idprovider/v1/auth/identitytoken`
    
   access_token=`echo $iam_json_data | jq .access_token | cut -d\" -f2`
   echo "access_token is $access_token"
    
    
   pod=`oc get pods |grep ibm-cp4a-operator | cut -d' ' -f1`
   oc cp $BACKUP_DATA/platform-oidc-registration.json  $pod:/tmp/
    
    
   echo -e "\nRun OIDC Registration API on the restored cluster ..."
   oc  rsh $pod  curl -i -k -X POST --header "Authorization: Bearer $access_token" \
              --header "Content-Type: application/json" \
              --data "@/tmp/platform-oidc-registration.json"\
              https://$iam_cluster_address:443/idprovider/v1/auth/registration
    
   echo -e "\nDone."
   
   
   
   
   
   

1. oc scale deployment ibm-bts-cp4ba-bts-316-deployment --replicas=0 
   
   oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1
   export PX_USER=$(cat /etc/superuser-secret/username)
   export PX_PASSWORD=$(cat /etc/superuser-secret/password)
   psql -U postgres -f /var/lib/postgresql/data/bk_btsdb.sql
   exit
   
   oc scale deployment ibm-bts-cp4ba-bts-316-deployment --replicas=1