1. Introduction
Previous blog Sample of Backup and Restore Business Automation Workflow Environment for CP4BA 22.0.2 on NFS Storage describes how to backup and restore Business Automation Workflow in IBM Cloud Pak for Business Automation 22.0.2 environment to a different openshift environment. It describes procedures step by step, but those steps are complex if execute them manually.

In this blog, it introduces a series of scripts to automate the backup and restore procedure, it could reduce the complexity because most steps would be implemented by scripts.

The CP4BA deployment may be different in different user’s environment, so the backup and restore method for that environment maybe some different from others. In below section, it is using following configuration and method to backup and restore:

• Use Offline backup method. i.e. stop related pods before backup, so that there is no new data would be generated during backup, and the backup data could be in consistency state.
• The deployment is using NFS as storage, so the backup and restore method here based on the file system.
• The deployment is using dynamic provision.
• The deployment is BAW enterprise pattern for CP4BA 22.0.2 with Business Automation Workflow Capability and Content Platform Engine Capability.
• The deployment is using Postgresql single server as database server, and there is not configured JDBC over SSL.
• In this blog, would backup BTS data to S3 compatible storage.

The script in this blog is the automation implementation for above configuration, other different environments may have different requirements and this automation may not be suitable on those environments, but this automation provides the core functionality and could be customized for those environments. Please note, if applied to production environment, it’s better to refine those scripts to add more logic for fault detection and error handling, and make full testing for your scripts.

2. About the automation implementation

This automation implementation includes following scripts (the last section in this blog shows the implementation for each script):

• br_env
• br_pv.sh
• br_backup.sh
• br_restore.sh

3. How to run Backup automation

1. Download and copy all files (br_*) to a directory, such as /home/backup. This directory would be used to store all backup files and data.
2. Update the variable in br_env file. For example:

   # The namespace used to deploy BAW
   NS=bawent
   
   # The icp4acluster name, defined in BAW CR file: metadata.name
   ICP4ACLUSTER_NAME=icp4adeploy
   
   # The Account info for S3 compatible storage
   S3_ACCESS_KEY_ID=<access key>
   S3_ACCESS_SECRET_KEY=<access secret key>
   
   # The PV folder on primary environment.
   SOURCE_PV_DIR=/home/pv/2202
   
   # The PV folder on secondary environment.
   TARGET_PV_DIR=/home/pv/2202
   
   # The folder used to store the backup files and data, on both primary environment and secondary environment.
   BACKUP_DIR=/home/backup
   
   BACKUP_PV_DIR=$BACKUP_DIR/pvfiles
   BACKUP_DATA=$BACKUP_DIR/data

1. Execute br_backup.sh. The output would be:

• • In the directory $BACKUP_PV_DIR: store the backup PV files.
  • In the directory $BACKUP_DATA: store the retrieved secrets, data.

2. Execute databases backup using database commands.
3. Backup the CR file and the secret files associated to CR file.

4. How to run Restore automation

1. Copy all backup files and data to the secondary environment. For example, copy all files in /home/backup from primary environment to same folder on the secondary environment.

1. Execute br_restore.sh. The necessary PVC, PV, secret would be restored.
2. Restore databases.
3. Restore secrets used by CR.
4. Modify CR as following:

5. Deploy CR.

5. Scripts Implementation

 a. br_env

# The namespace used to deploy BAW
NS=bawent

# The icp4acluster name, defined in BAW CR file: metadata.name
ICP4ACLUSTER_NAME=icp4adeploy

# The Account info for S3 compatible storage
S3_ACCESS_KEY_ID=<access key>
S3_ACCESS_SECRET_KEY=<access secret key>

# The PV folder on primary environment.
SOURCE_PV_DIR=/home/pv/2202

# The PV folder on secondary environment.
TARGET_PV_DIR=/home/pv/2202

# The folder used to store the backup files and data, on both primary environment and secondary environment.
BACKUP_DIR=/home/backup

BACKUP_PV_DIR=$BACKUP_DIR/pvfiles
BACKUP_DATA=$BACKUP_DIR/data

b. br_pv.sh

#!/bin/sh

source ./br_env

pv_backup_for_all() {
    if [ ! -d $BACKUP_PV_DIR ] ; then
        mkdir -p $BACKUP_PV_DIR
    fi

    oc get pvc -n $NS --no-headers=true | while read each
    do
        pvc=`echo $each | awk '{ print $1 }'`
        pv=`echo $each | awk '{ print $3 }'`
        
        if [  -d "$SOURCE_PV_DIR/$NS-$pvc-$pv" ]
        then
            echo "  backup pv $pv for pvc $pvc"
            mkdir -p $BACKUP_PV_DIR/$pvc
            cp -r -a $SOURCE_PV_DIR/$NS-$pvc-$pv/.  $BACKUP_PV_DIR/$pvc
        else
            echo "  NOT FOUND for $pvc !"
        fi
    done

    cd 
    echo -e "\n  compress the PV backup files  into $BACKUP_DIR/pv.tgz ..."
    cd $BACKUP_DIR
    tar cpzf /$BACKUP_DIR/pv.tgz  ./pvfiles/*

    rm -rf  $BACKUP_PV_DIR
    #echo "cd /home/backup ;  rm -rf /home/pv/2103/* ;  tar xpzf pv.tar.gz"
}


get_pvc_yaml_for_all() {
# wget  https://github.com/mikefarah/yq/releases/download/v4.9.5/yq_linux_amd64.tar.gz

pvc_file="pvc_all.yaml"

oc get pvc -n $NS --no-headers=true | while read each
do
    pvc=`echo $each | awk '{ print $1 }'`
	
	kubectl get pvc $pvc -o yaml \
      | yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' -  >> $BACKUP_DATA/$pvc_file
	
    echo "---" >> $BACKUP_DATA/$pvc_file
done
}


get_pvc_yaml() {
# wget  https://github.com/mikefarah/yq/releases/download/v4.9.5/yq_linux_amd64.tar.gz

pvc_file="pvc.yaml"

oc get pvc -n $NS --no-headers=true | while read each
do
    pvc=`echo $each | awk '{ print $1 }'`

    pvc_backup=false
	
    case $pvc in
        *"cpe-filestore"*)  pvc_backup=true ;;
        *"datadir-zen-metastoredb-"*)  pvc_backup=true ;;
        *"icn-cfgstore"*)  pvc_backup=true ;;
        *"baw-jms-data-"*)  pvc_backup=true ;;
        *"jms-pvc-"*)  pvc_backup=true ;;
        *) continue ;;
    esac

    if [ "$pvc_backup" = true ] ; then
        echo "  backup pvc yaml for $pvc"
        kubectl get pvc $pvc -o yaml \
          | yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' -  >> $BACKUP_DATA/$pvc_file
        echo "---" >> $BACKUP_DATA/$pvc_file
    fi
done
}


pv_restore() {
    if [ ! -d $BACKUP_PV_DIR ] ; then
        echo "Not found backup files! "
        exit
    fi

    oc get pvc -n $NS --no-headers=true | while read each
    do
        pvc=`echo $each | awk '{ print $1 }'`
      	pv=`echo $each | awk '{ print $3 }'`
       	# echo "$pvc : $pv "
   		
        case $pvc in
            *"cpe-filestore"*)  echo "restore $pvc" ;;
		    *"datadir-zen-metastoredb-"*)  echo "restore $pvc" ;;
		    *"icn-cfgstore"*)  echo "restore $pvc" ;;
		    *"baw-jms-data-"*)  echo "restore $pvc" ;;
            *"jms-pvc-"*)  echo "restore $pvc" ;;
            *) continue ;;
        esac
        if [  -d "$BACKUP_PV_DIR/$pvc" ]
      	then
   			echo "  cp -r -a $BACKUP_PV_DIR/$pvc/.  $TARGET_PV_DIR/$NS-$pvc-$pv/ "
            cp -r -a $BACKUP_PV_DIR/$pvc/*  $TARGET_PV_DIR/$NS-$pvc-$pv/
   		else
   		    echo "NOT FOUND for $pvc"
   		fi
    done
}


select_menu() {
    echo "Following operations are available : "
    echo "  0) backup PVC yaml to $BACKUP_DATA"
    echo "  1) backup all PV files to $BACKUP_PV_DIR"
    echo "  2) restore selected PV files from  $BACKUP_PV_DIR  to  $TARGET_PV_DIR "
    read -n 1 -p "Select 0/1/2 : " ans;
	echo ""
	
    case $ans in
        0)  get_pvc_yaml; get_pvc_yaml_for_all;;
        1)  pv_backup_for_all;;
        2)  pv_restore;;
        *)  echo "wrong input"; exit;;
    esac
}


if [ -z "$1" ] 
then
    select_menu
else
    ARRAY=(pv_backup_for_all   get_pvc_yaml_for_all   get_pvc_yaml   pv_restore)
    
    if echo "${ARRAY[@]}" | grep -w "$1" &>/dev/null; then
        echo -e "\n  calling  $1"
        eval $1
    else
        echo "  Please input one of following parameters: ${ARRAY[@]}"
        echo "  wrong input, exit."
        exit -1
    fi
fi

c. br_backup.sh

#!/bin/sh

source ./br_env

echo -e "\n...TO BACKUP SOURCE ENVIRONMENT...\n"

echo -e "\nChecking if the required commands existed ..."

if [! command -v jq &> /dev/null]; then
    echo "  jq was not installed!"
    exit -1
else
	echo "  jq was installed "
fi

if [! command -v yq &> /dev/null]; then
    echo "  yq was not installed!"
    exit -1
else
	echo "  yq was installed "
fi


echo -e "\nAll backup files and data would be collected into folder $BACKUP_DIR"
if [ ! -d $BACKUP_DIR ] ; then
    echo "Not found $BACKUP_DIR, create it "
	mkdir -p $BACKUP_DIR 
fi
if [ ! -d $BACKUP_DATA ] ; then
    echo "Not found $BACKUP_DATA, create it "
    mkdir -p $BACKUP_DATA
fi
if [ ! -d $BACKUP_PV_DIR ] ; then
    echo "Not found $BACKUP_PV_DIR, create it "
    mkdir -p $BACKUP_PV_DIR
fi


echo -e "\nChecking if has already login openshift or not ..."
ret=$( oc whoami 2>&1 )
if [[ $ret =~ "error" ]] ; then
   echo "Please login firstly ! e.g. 'oc login -u kubeadmin'"
   echo ""
   exit -1
fi


echo -e "\nGet uid for namespace $NS ..."
uid=`oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1 `
echo $uid >  "$BACKUP_DATA/src_uid"


echo -e "\nBackup secrets ..."

echo "  backup secret $ICP4ACLUSTER_NAME-cpe-oidc-secret ..."
kubectl get secret "$ICP4ACLUSTER_NAME-cpe-oidc-secret" -o yaml \
      | yq eval 'del(.metadata.creationTimestamp, .metadata.ownerReferences, .metadata.resourceVersion, .metadata.uid)' -  > "$BACKUP_DATA/src_cpe-oidc-secret.yaml"

echo "  backup secret admin-user-details ..."
kubectl get secret admin-user-details -o yaml \
      | yq eval 'del(.metadata.annotations, .metadata.creationTimestamp, .metadata.ownerReferences, .metadata.resourceVersion, .metadata.uid)' -   > "$BACKUP_DATA/src_admin-user-details.yaml"

echo "  backup secret ibm-bts-cnpg-$NS-cp4ba-bts-app ..."
kubectl get secret "ibm-bts-cnpg-$NS-cp4ba-bts-app" -o yaml \
      | yq eval 'del(.metadata.annotations, .metadata.creationTimestamp, .metadata.ownerReferences, .metadata.resourceVersion, .metadata.uid)' -  >  "$BACKUP_DATA/ibm-bts-cnpg-$NS-cp4ba-bts-app.yaml"


echo -e "\nScale down operators, deployment and statefulset to 0 ..."

echo "  stop operators ..." 
oc scale deploy ibm-cp4a-operator --replicas=0
oc scale deploy ibm-pfs-operator  --replicas=0
oc scale deploy ibm-content-operator  --replicas=0

# echo "  stop deployment & statefulset ..."
for i in `oc get deploy -o name |grep $ICP4ACLUSTER_NAME`; do echo "  stop $i" ; oc scale $i --replicas=0; done
for i in `oc get sts -o name |grep $ICP4ACLUSTER_NAME`; do echo "  stop $i" ; oc scale $i --replicas=0; done

echo "  stop sts zen-metastoredb" ;  oc scale sts  zen-metastoredb --replicas=0

echo -e "\n>waiting Pods Scaling down to be done"
sleep 15
while oc get pods | grep "Terminating"
do
    echo -e "\n=>waiting Pods scaling down to be done"
    sleep 15
done


echo -e "\nBackup BTS data ..."

cat << EOF > br_bts_backup.yaml
apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Backup
metadata:
  name: bts-$NS-backup
spec:
  cluster:
    name: ibm-bts-cnpg-$NS-cp4ba-bts
EOF

oc apply -f br_bts_backup.yaml

echo -e "  waiting backup status to be completed"
sleep 15
while ! (oc get backup -n "$NS"  "bts-$NS-backup"  -o jsonpath='{.status.phase}{"\n"}' | grep -q "completed")
do
    sleep 15
done


echo -e "\nBackup PVC definitions ..."
./br_pv.sh  get_pvc_yaml                   
./br_pv.sh  get_pvc_yaml_for_all            


echo -e "\nBackup all PV files ..."
./br_pv.sh  pv_backup_for_all              


echo -e "\nNext Steps:"
echo "  a. Backup databases."
echo "  b. Copy $BACKUP_DIR to target Openshift with same folder."


echo -e "\nBackup Done.\n"

d. br_restore.sh

#!/bin/sh

source ./br_env

echo -e "\n...TO RESTORE TARGET ENVIRONMENT...\n"

echo -e "\nChecking if the required commands existed ..."

if [! command -v jq &> /dev/null]; then
    echo "  jq was not installed!"
    exit -1
else
	echo "  jq was installed "
fi

if [! command -v yq &> /dev/null]; then
    echo "  yq was not installed!"
    exit -1
else
	echo "  yq was installed "
fi


echo -e "\nChecking backup folders ..."
if [ ! -d $BACKUP_DIR ] ; then
    echo "Not found $BACKUP_DIR! "
	exit -1
fi
if [ ! -d $BACKUP_DATA ] ; then
    echo "Not found $BACKUP_DATA! "
    exit -1
fi
if [ ! -d $BACKUP_PV_DIR ] ; then
    echo "Not found $BACKUP_PV_DIR, create it!"
    mkdir -p $BACKUP_PV_DIR
fi


echo -e "\nChecking if has already login or not ..."
ret=$( oc whoami 2>&1 )
if [[ $ret =~ "error" ]] ; then
   echo "Please login firstly ! e.g. 'oc login -u kubeadmin'"
   echo ""
   exit -1
fi


echo -e "\nGet uid for namespace $NS ..."
uid=`oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1 `
echo $uid >  "$BACKUP_DATA/tgt_uid"

echo -e "\nApply src_admin-user-details.yaml ..."
oc apply  -f  "$BACKUP_DATA/src_admin-user-details.yaml"

echo -e "\nApply src_cpe-oidc-secret.yaml ..."
oc apply  -f  "$BACKUP_DATA/src_cpe-oidc-secret.yaml"

echo -e "\nApply ibm-bts-cnpg-bawent-cp4ba-bts-app.yaml ..."
oc apply  -f  "$BACKUP_DATA/ibm-bts-cnpg-bawent-cp4ba-bts-app.yaml" 


echo -e "\nCreate secret s3-credentials ..."
kubectl delete secret s3-credentials
kubectl create secret generic s3-credentials \
  --from-literal=ACCESS_KEY_ID=$S3_ACCESS_KEY_ID \
  --from-literal=ACCESS_SECRET_KEY=$S3_ACCESS_SECRET_KEY


echo -e "\nRestore PVC ..."
oc apply  -f  "$BACKUP_DATA/pvc.yaml"


echo -e "\nUncompress pv.tgz ..."
cd $BACKUP_DIR ; rm -rf $BACKUP_PV_DIR/* 
tar xpzf  $BACKUP_DIR/pv.tgz


echo -e "\nChange uid for PV files ..."
src_uid=`cat  $BACKUP_DATA/src_uid`
tgt_uid=`cat  $BACKUP_DATA/tgt_uid`
cd $BACKUP_PV_DIR ; find ./  -uid  $src_uid  -exec chown $tgt_uid:root {} \;


echo -e "\nRestore PV files ..."
cd $BACKUP_DIR
$BACKUP_DIR/br_pv.sh  pv_restore


echo -e "\nNext Steps:"
echo "  a. restore databases"
echo "  b. restore (or apply) secrets used for CR"
echo "  c. modify CR as following:"
echo "       spec.shared_configuration.sc_content_initialization: false "
echo "       spec.shared_configuration.sc_content_verification: false "
echo "       spec.shared_configuration.sc_deployment_hostname_suffix: <correct name>"
echo "     if necessary, add section for bts recovery."
echo "  d. deploy CR, waiting deployment to be done."
echo "  "
echo "  "