Introduction
Previous blog "Sample of Backup and Restore Business Automation Workflow Environment for CP4BA 21.0.3 on NFS Storage" describes how to backup and restore Business Automation Workflow in IBM Cloud Pak for Business Automation 21.0.3 environment to a different openshift environment. It describes procedures step by step, but those steps are complex if execute them manually.
In this blog, it introduces a series of scripts to automate the backup and restore procedure, it could reduce the complexity because most steps would be implemented by scripts.
The CP4BA deployment may be different in different user’s environment, so the backup and restore method for that environment maybe some different from others. In below section, it is using following configuration and method to backup and restore:
- Use Offline backup method. i.e. stop related pods before backup, so that there is no new data would be generated during backup, and the backup data could be in consistency state.
- The deployment is using NFS as storage class, so the backup and restore method here based on the file system.
- The deployment is using dynamic provision.
- The deployment is BAW enterprise pattern for CP4BA 21.0.3 with Business Automation Workflow Capability and Content Platform Engine Capability.
- The deployment is using DB2 single server as database server, and there is not configured JDBC over SSL.
The script in this blog is the automation implementation for above configuration, other different environments may have different requirements and this automation may not be suitable on those environments, but this automation provides the core functionality and could be customized for those environments.
About the automation implementation
This automation implementation includes following scripts (the last section in this blog shows the implementation for each script):
- br_env
- br_pv.sh
- br_backup.sh
- br_restore.sh
- br_restore_bts.sh
- br_restore_iam.sh
How to run Backup automation
Prepare Before Backup
- Download and copy all files (br_*) to a directory, such as /home/backup. This directory would be used to store all backup files and data.
- Update the variable in br_env file. For example:
# The namespace used to deploy BAW
NS=bawent
# The icp4acluster name, defined in CR file
ICP4ACLUSTER_NAME=demo
# The PV folder on primary environment.
SOURCE_PV_DIR=/home/pv/2103
# The PV folder on secondary environment.
TARGET_PV_DIR=/home/pv/2103
# The folder used to store the backup files and data, on both primary environment and secondary environment.
BACKUP_DIR=/home/backup
BACKUP_PV_DIR=$BACKUP_DIR/pvfiles
BACKUP_DATA=$BACKUP_DIR/data
How to run backup
- Execute
br_backup.sh. The output would be:
- In the directory
$BACKUP_PV_DIR: store the backup PV files.
- In the directory
$BACKUP_DATA: store the retrieved secrets, data.
- Execute databases backup using database commands.
- Backup the CR file and the secret files associated to CR file.
How to run Restore automation
Prepare Before Restore
- Copy all backup files and data to the secondary environment. For example, copy all files in /home/backup from primary environment to same folder on the secondary environment.
How to run Restore
- Execute
br_restore.sh. The necessary PVC, PV, secret would be restored.
- Restore databases.
- Restore secrets used by CR.
- Modify CR as following:
spec.shared_configuration.sc_content_initialization: falsespec.shared_configuration.sc_content_verification: falsespec.shared_configuration.sc_deployment_hostname_suffix: <correct name>
- Deploy CR.
- During deployment, execute
br_restore_iam.sh.
Please note, this step was improved in the next release, the br_restore_iam.sh may be dropped due to design improvement in later release.
- Wait the deployment to be done.
- Restore BTS, execute
br_restore_bts.sh.
Scripts Implementation
br_env
# The namespace used to deploy BAW
NS=bawent
# The icp4acluster name, defined in CR file
ICP4ACLUSTER_NAME=demo
# The PV folder on primary environment.
SOURCE_PV_DIR=/home/pv/2103
# The PV folder on secondary environment.
TARGET_PV_DIR=/home/pv/2103
# The folder used to store the backup files and data, on both primary environment and secondary environment.
BACKUP_DIR=/home/backup
BACKUP_PV_DIR=$BACKUP_DIR/pvfiles
BACKUP_DATA=$BACKUP_DIR/data
br_pv.sh
#!/bin/sh
source ./br_env
pv_backup_for_all() {
if [ ! -d $BACKUP_PV_DIR ] ; then
mkdir -p $BACKUP_PV_DIR
fi
oc get pvc -n $NS --no-headers=true | while read each
do
pvc=`echo $each | awk '{ print $1 }'`
pv=`echo $each | awk '{ print $3 }'`
if [ -d "$SOURCE_PV_DIR/$NS-$pvc-$pv" ]
then
echo " backup pv $pv for pvc $pvc"
mkdir -p $BACKUP_PV_DIR/$pvc
cp -r -a $SOURCE_PV_DIR/$NS-$pvc-$pv/. $BACKUP_PV_DIR/$pvc
else
echo " NOT FOUND for $pvc !"
fi
done
cd
echo -e "\n compress the PV backup files into $BACKUP_DIR/pv.tgz ..."
cd $BACKUP_DIR
tar cpzf /$BACKUP_DIR/pv.tgz ./pvfiles/*
rm -rf $BACKUP_PV_DIR
}
get_pvc_yaml_for_all() {
pvc_file="pvc_all.yaml"
oc get pvc -n $NS --no-headers=true | while read each
do
pvc=`echo $each | awk '{ print $1 }'`
kubectl get pvc $pvc -o yaml \
| yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' - >> $BACKUP_DATA/$pvc_file
echo "---" >> $BACKUP_DATA/$pvc_file
done
}
get_pvc_yaml() {
pvc_file="pvc.yaml"
oc get pvc -n $NS --no-headers=true | while read each
do
pvc=`echo $each | awk '{ print $1 }'`
pvc_backup=false
case $pvc in
*"cpe-filestore"*) pvc_backup=true ;;
*"datadir-zen-metastoredb-"*) pvc_backup=true ;;
*"icn-cfgstore"*) pvc_backup=true ;;
*"baw-jms-data-"*) pvc_backup=true ;;
*"jms-pvc-"*) pvc_backup=true ;;
*) continue ;;
esac
if [ "$pvc_backup" = true ] ; then
echo " backup pvc yaml for $pvc"
kubectl get pvc $pvc -o yaml \
| yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' - >> $BACKUP_DATA/$pvc_file
echo "---" >> $BACKUP_DATA/$pvc_file
fi
done
}
pv_restore() {
if [ ! -d $BACKUP_PV_DIR ] ; then
echo "Not found backup files! "
exit
fi
oc get pvc -n $NS --no-headers=true | while read each
do
pvc=`echo $each | awk '{ print $1 }'`
pv=`echo $each | awk '{ print $3 }'`
case $pvc in
*"cpe-filestore"*) echo "restore $pvc" ;;
*"datadir-zen-metastoredb-"*) echo "restore $pvc" ;;
*"icn-cfgstore"*) echo "restore $pvc" ;;
*"baw-jms-data-"*) echo "restore $pvc" ;;
*"jms-pvc-"*) echo "restore $pvc" ;;
*) continue ;;
esac
if [ -d "$BACKUP_PV_DIR/$pvc" ]
then
echo " cp -r -a $BACKUP_PV_DIR/$pvc/. $TARGET_PV_DIR/$NS-$pvc-$pv/ "
cp -r -a $BACKUP_PV_DIR/$pvc/* $TARGET_PV_DIR/$NS-$pvc-$pv/
else
echo "NOT FOUND for $pvc"
fi
done
}
select_menu() {
echo "Following operations are available : "
echo " 0) backup PVC yaml to $BACKUP_DATA"
echo " 1) backup all PV files to $BACKUP_PV_DIR"
echo " 2) restore selected PV files from $BACKUP_PV_DIR to $TARGET_PV_DIR "
read -n 1 -p "Select 0/1/2 : " ans;
echo ""
case $ans in
0) get_pvc_yaml; get_pvc_yaml_for_all;;
1) pv_backup_for_all;;
2) pv_restore;;
*) echo "wrong input"; exit;;
esac
}
if [ -z "$1" ]
then
select_menu
else
ARRAY=(pv_backup_for_all get_pvc_yaml_for_all get_pvc_yaml pv_restore)
if echo "${ARRAY[@]}" | grep -w "$1" &>/dev/null; then
echo -e "\n calling $1"
eval $1
else
echo " Please input one of following parameters: ${ARRAY[@]}"
echo " wrong input, exit."
exit -1
fi
fi
br_backup.sh
#!/bin/sh
source ./br_env
echo -e "\n...TO BACKUP SOURCE ENVIRONMENT...\n"
echo -e "\nChecking if the required commands existed ..."
if [! command -v jq &> /dev/null]; then
echo " jq was not installed!"
exit -1
else
echo " jq was installed "
fi
if [! command -v yq &> /dev/null]; then
echo " yq was not installed!"
exit -1
else
echo " yq was installed "
fi
echo -e "\nAll backup files and data would be collected into folder $BACKUP_DIR"
if [ ! -d $BACKUP_DIR ] ; then
echo "Not found $BACKUP_DIR, create it "
mkdir -p $BACKUP_DIR
fi
if [ ! -d $BACKUP_DATA ] ; then
echo "Not found $BACKUP_DATA, create it "
mkdir -p $BACKUP_DATA
fi
if [ ! -d $BACKUP_PV_DIR ] ; then
echo "Not found $BACKUP_PV_DIR, create it "
mkdir -p $BACKUP_PV_DIR
fi
echo -e "\nChecking if has already login or not ..."
ret=$( oc whoami 2>&1 )
if [[ $ret =~ "error" ]] ; then
echo "Please login firstly ! e.g. 'oc login -u kubeadmin'"
echo ""
exit -1
fi
echo -e "\nGet uid for namespace $NS ..."
uid=`oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1 `
echo $uid > "$BACKUP_DATA/src_uid"
echo -e "\nBackup secrets ..."
echo " backup WLP_CLIENT_ID in secret platform-oidc-credentials ..."
oc -n ibm-common-services get secret platform-oidc-credentials -o jsonpath='{.data.WLP_CLIENT_ID}' | base64 -d > "$BACKUP_DATA/src_WLP_CLIENT_ID"
echo " backup WLP_CLIENT_SECRET in secret platform-oidc-credentials ..."
oc -n ibm-common-services get secret platform-oidc-credentials -o jsonpath='{.data.WLP_CLIENT_SECRET}' | base64 -d > "$BACKUP_DATA/src_WLP_CLIENT_SECRET"
echo " backup secret admin-user-details ..."
kubectl get secret admin-user-details -o yaml \
| yq eval 'del(.metadata.annotations, .metadata.creationTimestamp, .metadata.ownerReferences, .metadata.resourceVersion, .metadata.uid)' - > "$BACKUP_DATA/src_admin-user-details.yaml"
echo -e "\nScale down deployment and statefulset to 0 ..."
echo " stop deployment ibm-cp4a-operator" ; oc scale deploy ibm-cp4a-operator --replicas=0
for i in `oc get deploy -o name |grep $ICP4ACLUSTER_NAME`; do echo " stop $i" ; oc scale $i --replicas=0; done
for i in `oc get sts -o name |grep $ICP4ACLUSTER_NAME`; do echo " stop $i" ; oc scale $i --replicas=0; done
echo " stop sts zen-metastoredb" ; oc scale sts zen-metastoredb --replicas=0
echo -e "\nBackup BTS data ..."
rm -f ./backup_bts.sh
echo 'export PX_USER=$(cat /etc/superuser-secret/username)' >> backup_bts.sh
echo 'export PX_PASSWORD=$(cat /etc/superuser-secret/password)' >> backup_bts.sh
echo 'pg_dump -d BTSDB -U postgres -Fp -c -C --if-exists -f /var/lib/postgresql/data/bk_btsdb.sql ' >> backup_bts.sh
echo -e "\n backup data for ibm-bts-cnpg-bawent-cp4ba-bts-1..."
oc cp backup_bts.sh ibm-bts-cnpg-bawent-cp4ba-bts-1:/var/lib/postgresql/data/
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 chmod 755 /var/lib/postgresql/data/backup_bts.sh
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 bash -c /var/lib/postgresql/data/backup_bts.sh
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 ls -l /var/lib/postgresql/data/
oc cp ibm-bts-cnpg-bawent-cp4ba-bts-1:/var/lib/postgresql/data/bk_btsdb.sql $BACKUP_DATA/src_btsdb_1.sql
rm -f ./backup_bts.sh
echo -e "\nBackup PVC definitions ..."
$BACKUP_DIR/br_pv.sh get_pvc_yaml
$BACKUP_DIR/br_pv.sh get_pvc_yaml_for_all
echo -e "\nBackup all PV files ..."
$BACKUP_DIR/br_pv.sh pv_backup_for_all
echo -e "\nNext, backup databases. "
echo -e "\nNext, Copy $BACKUP_DIR to target VMs with same folder."
echo -e "\nBackup Done.\n"
br_restore.sh
#!/bin/sh
source ./br_env
echo -e "\n...TO BACKUP SOURCE ENVIRONMENT...\n"
echo -e "\nChecking if the required commands existed ..."
if [! command -v jq &> /dev/null]; then
echo " jq was not installed!"
exit -1
else
echo " jq was installed "
fi
if [! command -v yq &> /dev/null]; then
echo " yq was not installed!"
exit -1
else
echo " yq was installed "
fi
echo -e "\nChecking backup folders ..."
if [ ! -d $BACKUP_DIR ] ; then
echo "Not found $BACKUP_DIR! "
exit -1
fi
if [ ! -d $BACKUP_DATA ] ; then
echo "Not found $BACKUP_DATA! "
exit -1
fi
if [ ! -d $BACKUP_PV_DIR ] ; then
echo "Not found $BACKUP_PV_DIR, create it!"
mkdir -p $BACKUP_PV_DIR
fi
echo -e "\nChecking if has already login or not ..."
ret=$( oc whoami 2>&1 )
if [[ $ret =~ "error" ]] ; then
echo "Please login firstly ! e.g. 'oc login -u kubeadmin'"
echo ""
exit -1
fi
echo -e "\nGet uid for namespace $NS ..."
uid=`oc describe project $NS | grep uid-range | cut -d"=" -f2 | cut -d"/" -f1 `
echo $uid > "$BACKUP_DATA/tgt_uid"
echo -e "\nApply src_admin-user-details.yaml ..."
oc apply -f "$BACKUP_DATA/src_admin-user-details.yaml"
echo -e "\nRestore PVC ..."
oc apply -f "$BACKUP_DATA/pvc.yaml"
echo -e "\nUncompress pv.tgz ..."
cd $BACKUP_DIR ; rm -rf $BACKUP_PV_DIR/*
tar xpzf $BACKUP_DIR/pv.tgz
echo -e "\nChange uid for PV files ..."
src_uid=`cat $BACKUP_DATA/src_uid`
tgt_uid=`cat $BACKUP_DATA/tgt_uid`
cd $BACKUP_PV_DIR ; find ./ -uid $src_uid -exec chown $tgt_uid:root {} \;
echo -e "\nRestore PV files ..."
cd $BACKUP_DIR
$BACKUP_DIR/br_pv.sh pv_restore
echo -e "\nNext Steps:"
echo " a. restore databases"
echo " b. restore secrets used by CR"
echo " c. modify CR as following:"
echo " spec.shared_configuration.sc_content_initialization: false "
echo " spec.shared_configuration.sc_content_verification: false "
echo " spec.shared_configuration.sc_deployment_hostname_suffix: <correct name>"
echo " d. deploy CR"
echo " e. During deployment, run br_restore_iam.sh "
echo " f. WAITING deployment to be done"
echo " g. restore BTS, run br_restore_bts.sh"
echo " "
echo " "
br_restore_iam.sh
#!/bin/sh
source ./br_env
echo -e "\nRestore IAM Registration data ..."
echo -e "\nWaiting configmap registration-json to be created ... "
while true
do
if oc -n ibm-common-services get configmap | grep -q registration-json ; then
echo " configmap registration-json was created! waiting 180 sec ..."
sleep 180
break
else
sleep 15
fi
done
echo -e "\nGet & update platform-oidc-registration.json"
oc -n ibm-common-services get configmap registration-json -o jsonpath='{.data.*}' > $BACKUP_DATA/platform-oidc-registration.json
WID=`cat $BACKUP_DATA/src_WLP_CLIENT_ID`
jq '."client_id" = "'"$WID"'"' $BACKUP_DATA/platform-oidc-registration.json > tmp && mv tmp $BACKUP_DATA/platform-oidc-registration.json
WST=`cat $BACKUP_DATA/src_WLP_CLIENT_SECRET`
jq '."client_secret" = "'"$WST"'"' $BACKUP_DATA/platform-oidc-registration.json > tmp && mv tmp $BACKUP_DATA/platform-oidc-registration.json
echo -e "\nGet access_token ..."
iam_user=`oc -n ibm-common-services get secret ibm-iam-bindinfo-platform-auth-idp-credentials -o jsonpath='{.data.admin_username}' | base64 -d `
iam_pwd=`oc -n ibm-common-services get secret ibm-iam-bindinfo-platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d`
iam_cluster_address=`oc get route -n ibm-common-services cp-console -o jsonpath={.spec.host}`
iam_json_data=`curl -k -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' \
-d "grant_type=password&username=$iam_user&password=$iam_pwd&scope=openid" \
https://$iam_cluster_address/idprovider/v1/auth/identitytoken`
access_token=`echo $iam_json_data | jq .access_token | cut -d\" -f2`
echo "access_token is $access_token"
pod=`oc get pods |grep ibm-cp4a-operator | cut -d' ' -f1`
oc cp $BACKUP_DATA/platform-oidc-registration.json $pod:/tmp/
echo -e "\nRun OIDC Registration API on the restored cluster ..."
oc rsh $pod curl -i -k -X POST --header "Authorization: Bearer $access_token" \
--header "Content-Type: application/json" \
--data "@/tmp/platform-oidc-registration.json"\
https://$iam_cluster_address:443/idprovider/v1/auth/registration
echo -e "\nDone."
br_restore_bts.sh
#!/bin/sh
source ./br_env
echo -e "\nRestore BTS data ..."
oc scale deployment ibm-bts-cp4ba-bts-316-deployment --replicas=0 # stop BTS server
rm -f ./backup_bts.sh
echo 'export PX_USER=$(cat /etc/superuser-secret/username)' >> backup_bts.sh
echo 'export PX_PASSWORD=$(cat /etc/superuser-secret/password)' >> backup_bts.sh
echo 'pg_dump -d BTSDB -U postgres -Fp -c -C --if-exists -f /var/lib/postgresql/data/tgt_btsdb.sql.bak ' >> backup_bts.sh
echo 'psql -U postgres -f /var/lib/postgresql/data/src_btsdb.sql' >> backup_bts.sh
echo -e "\n restore data for ibm-bts-cnpg-bawent-cp4ba-bts-1..."
oc cp backup_bts.sh ibm-bts-cnpg-bawent-cp4ba-bts-1:/var/lib/postgresql/data/
oc cp $BACKUP_DATA/src_btsdb_1.sql ibm-bts-cnpg-bawent-cp4ba-bts-1:/var/lib/postgresql/data/src_btsdb.sql
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 chmod 755 /var/lib/postgresql/data/backup_bts.sh
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 bash -c /var/lib/postgresql/data/backup_bts.sh
oc rsh ibm-bts-cnpg-bawent-cp4ba-bts-1 ls -l /var/lib/postgresql/data/
oc cp ibm-bts-cnpg-bawent-cp4ba-bts-1:/var/lib/postgresql/data/tgt_btsdb.sql.bak $BACKUP_DATA/tgt_btsdb_1.sql.bak
rm -f ./backup_bts.sh
oc scale deployment ibm-bts-cp4ba-bts-316-deployment --replicas=2 # start BTS server
echo -e "\nDone."