Seamless Transitions: Cloud Pak for Data Storage Class Migration in Action
For Cloud Pak for Data clusters installed with incorrect storage classes, there is a potential risk of stability and performance issues. Reinstalling the cluster with the correct storage classes can be resource-intensive, especially for production environments with large data volumes.
Is there an alternative solution to address the problem of incorrect storage classes? The answer is yes. This article will showcase a practical approach to Cloud Pak for Data Storage Class Migration, enabling you to mitigate the issue without a full cluster re-installation.
Note:
Although this solution has been tested and validated across multiple environments, it is not an officially certified storage class migration method by IBM. Therefore, you would need to proceed at your own risk.
Migration context
Environment info
OCP: 4.14
CPD: 4.8.1
ODF: 4.14
Componenets: cpfs,cpd_platform,wkc,analyticsengine
Incorrect storage classes use case
When installing Cloud Pak for Data, the file storage class ocs-storagecluster-cephfs was used by the core database services (e.g. CoudhDb, ElasticSearch, RabbitMQ and Redis) of the Common Core Service component. The block storage class ocs-storagecluster-ceph-rbd should be used instead.
Here's the example about the CoudhDb persistent volumes which needs to be migrated from ocs-storagecluster-cephfs to ocs-storagecluster-ceph-rbd.
database-storage-wdp-couchdb-0 Bound pvc-e80583ab-5616-4fab-b68a-bef3dd9efd29 45Gi RWO ocs-storagecluster-cephfs 6d23h
database-storage-wdp-couchdb-1 Bound pvc-f869e03c-143a-46d1-b17a-9f6ae23a25da 45Gi RWO ocs-storagecluster-cephfs 6d23h
database-storage-wdp-couchdb-2 Bound pvc-eaa3bb34-6eb8-40f7-b34d-6869b0967ca6 30Gi RWO ocs-storagecluster-cephfs 6d23h
Note:
If there are other database services (e.g. Redis, RabbitMQ) of the Common Core Service which needs the storage class migration, this migration reference can still work. While, if you plan to do PVC size change during the migration, additional special settings maybe required. To lower the risk, it's recommended engaging IBM Support or Expert Lab team instead.
Migration in action
1 Pre-migration tasks
1.1 Have a cluster level backup
Backup your Cloud Pak for Data installation before the storage class migration. For details, see Backing up and restoring Cloud Pak for Data (https://www.ibm.com/docs/en/SSQNUZ_4.8.x/cpd/admin/backup_restore.html).
1.2 Have backup for the statefulsets relevant to the PV migration
1.Create a backup dir.
mkdir -p /opt/ibm/cpd_pv_migration
export CPD_PV_MIGRATION_DIR=/opt/ibm/cpd_pv_migration
2.Bakup for the CCS CR.
export PROJECT_CPD_INST_OPERANDS=<your Cloud Pak for Data instance namespace>
oc get ccs -o yaml -n ${PROJECT_CPD_INST_OPERANDS} > ${CPD_PV_MIGRATION_DIR}/ccs-cr.yaml
3.Bakup for CouchDB statefulsets.
oc get sts -n ${PROJECT_CPD_INST_OPERANDS} | grep couch | awk '{print $1}'| xargs oc get sts -o yaml -n ${PROJECT_CPD_INST_OPERANDS} > ${CPD_PV_MIGRATION_DIR}/sts-wdp-couchdb-bak.yaml
for p in $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep couchdb | awk '{print $1}') ;do oc get pvc $p -o yaml -n ${PROJECT_CPD_INST_OPERANDS} > ${CPD_PV_MIGRATION_DIR}/pvc-$p-bak.yaml;done
for p in $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep couchdb | awk '{print $3}') ;do oc get pv $p -o yaml -n ${PROJECT_CPD_INST_OPERANDS} > ${CPD_PV_MIGRATION_DIR}/pv-$p-bak.yaml;done
1.3 Mirror images (optional)
This is only required by the air-gapped environment.
1.3.1 Mirror the rhel-tools image
This image will be used for mounting both the existing persistent volume claims and the new ones for copying the data from the existing ones to the new ones .
- 1.Save the image in an internet connected machine.
podman pull registry.access.redhat.com/rhel7/rhel-tools:latest
podman save registry.access.redhat.com/rhel7/rhel-tools:latest -o rhel-tools.tar
podman load -i rhel-tools.tar
podman images | grep rhel-tools
podman login -u <username> -p <password> <target_registry> --tls-verify=false
podman tag 643870113757 <target_registry>/rhel7/rhel-tools:latest
podman push <target_registry>/rhel7/rhel-tools:latest --tls-verify=false
1.4 The permissions required for the migration is ready
It's recommended having the Openshift cluster administrator permissions ready for this migration.
1.5 A health check is made to ensure the cluster's readiness for upgrade.
The OpenShift cluster, persistent storage and Cloud Pak for Data platform and services are in healthy status.
Log into OCP and run below command.
Make sure all the cluster operators in AVAILABLE status. And not in PROGRESSING or DEGRADED status.
Run this command and make sure all nodes in Ready status.
Run this command and make sure all the machine configure pool are in healthy status.
- 2.Check Cloud Pak for Data status
Log onto bastion node, run this command in terminal and make sure the Lite and all the services' status are in Ready status.
cpd-cli manage login-to-ocp \
--username=${OCP_USERNAME} \
--password=${OCP_PASSWORD} \
--server=${OCP_URL}
cpd-cli manage get-cr-status --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS}
Run this command and make sure all pods healthy.
oc get po --no-headers --all-namespaces -o wide | grep -Ev '([[:digit:]])/\1.*R' | grep -v 'Completed'
Make sure the ODF cluster status is healthy and also has enough capacity for creating new persistent volumes to store a copy of the CouchDb data.
oc describe cephcluster ocs-storagecluster-cephcluster -n openshift-storage
1.6 Schedule a maintenance time window
This migration work requires down time. Send a heads-up to all end-users before this migration.
And it's recommended disabling the Cloud Pak for Data route just before starting the storage class migration in a production cluster. So that it can protect the storage class migration from the interruption.
Have a backup of the CPD route and place it in a safe place.
oc get route -n ${PROJECT_CPD_INST_OPERANDS} -o yaml > cpd_routes.yaml
Delete the CPD route
oc delete route -n ${PROJECT_CPD_INST_OPERANDS} -o yaml > cpd_routes.yaml
Note:
These migration steps need to be validated carefully in a testing cluster before applying it in a production one. Down time is expected during this migration.
2.1.Put CCS into maintenance mode
Put CCS into maintenance mode for preventing the migration work from being impacted by the operator reconciliation.
oc patch ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n ${PROJECT_CPD_INST_OPERANDS}
Make sure the CCS put into the maintenance mode successfully.
oc get ccs ccs-cr -n ${PROJECT_CPD_INST_OPERANDS}
2.2.Change the ReclaimPolicy to be "Retain" for the existing PVs (the ones with the wrong SC ocs-storagecluster-cephfs)
Patch the CouchDB PVs.
for p in $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep couchdb | awk '{print $3}') ;do oc patch pv $p -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' -n ${PROJECT_CPD_INST_OPERANDS};done
Make sure the ReclaimPolicy of the CouchDB PVs are changed to be "Retain".
oc get pv | grep -i couchdb
pvc-05493166-c0e2-4b67-b683-277ee23f51d6 30Gi RWO Retain Bound cpd/database-storage-wdp-couchdb-2 ocs-storagecluster-cephfs 17h
pvc-2fb1d306-1d39-4860-acb4-f04bcbd48dea 30Gi RWO Retain Bound cpd/database-storage-wdp-couchdb-0 ocs-storagecluster-cephfs 17h
pvc-6cc51d6e-d882-4abd-b50d-9c8d4dbf6276 30Gi RWO Retain Bound cpd/database-storage-wdp-couchdb-1 ocs-storagecluster-cephfs 17h
2.3 Migration for CouchDB
- Get old PVC name and volume name.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep -i wdp-couchdb
Sample output looks like this:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
database-storage-wdp-couchdb-0 Bound pvc-e8eed9f7-7bb6-4b5d-ab8d-21d49c4ecf35 30Gi RWO ocs-storagecluster-cephfs 89d
database-storage-wdp-couchdb-1 Bound pvc-14d811c3-b4d8-42c5-b6d9-1c3a44c25534 30Gi RWO ocs-storagecluster-cephfs 89d
database-storage-wdp-couchdb-2 Bound pvc-fb73ce1c-36b0-4358-a66d-9142bf0ce7b7 30Gi RWO ocs-storagecluster-cephfs 89d
- Note the mount path of the data volume
/opt/couchdb/data by checking the volumeMounts definition in wdp-couchdb sts yaml file.
volumeMounts:
- name: database-storage
mountPath: /opt/couchdb/data
- Make sure that the replicas of wdp-couchdb sts has been scaled down to zero.
oc scale sts wdp-couchdb -n ${PROJECT_CPD_INST_OPERANDS} --replicas=0
oc get sts -n ${PROJECT_CPD_INST_OPERANDS} | grep -i wdp-couchdb
2.3.2 Start a new temporary deployment using the rhel-tools image
oc -n ${PROJECT_CPD_INST_OPERANDS} create deployment sleep --image=registry.access.redhat.com/rhel7/rhel-tools -- tail -f /dev/null
Change the image registry URL in the command if needed.
2.3.3 Migration for the database-storage-wdp-couchdb-0 pvc
- Create a new PVC by referencing the database-storage-wdp-couchdb-0 pvc.
oc get pvc database-storage-wdp-couchdb-0 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-0-new.json
Specify a new name and the right storage class (ocs-storagecluster-ceph-rbd) for the new PVC
tmp=$(mktemp)
jq '.metadata.name = "database-storage-wdp-couchdb-0-new"' pvc-database-storage-wdp-couchdb-0-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-0-new.json
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-0-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-0-new.json
Create the new PVC.
oc apply -f pvc-database-storage-wdp-couchdb-0-new.json
Make sure the new PVC is created successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-0-new
Scale down the sleep deployment
oc scale deploy sleep --replicas=0
Wait until it's scaled down .
- Mount the old database-storage-wdp-couchdb-0 PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=old-claim --claim-name=database-storage-wdp-couchdb-0 --mount-path=/old-claim
- Mount the new database-storage-wdp-couchdb-0-new PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=new-claim --claim-name=database-storage-wdp-couchdb-0-new --mount-path=/new-claim
- Scale back the sleep deployment
oc scale deploy sleep --replicas=1
Wait until the sleep pod is up and running.
oc get pods | grep -i sleep
oc rsh $(oc get pod | grep sleep | awk '{print $1}')
- Migrate data to the new storage:
rsync -avxHAX --progress /old-claim/* /new-claim
Note:
Make sure the terminal session will not be closed or expired to protect the copy from the interruption during this step.
Ensure the number of files between the old-claim and the new-claim folder is same. For example:
sh-4.2$ cd old-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
sh-4.2$ cd ../new-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
- Remove the volume mounts from the sleep deployment
oc set volume deployment sleep --remove --confirm
- Patch the PV of the database-storage-wdp-couchdb-0-new PVC for chaning the ReclaimPolicy to be "Retain"
oc patch pv $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep wdp-couchdb-0-new | awk '{print $3}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' -n ${PROJECT_CPD_INST_OPERANDS}
Make sure the ReclaimPolicy was change to be "Retain" successfully.
oc get pv $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep wdp-couchdb-0-new | awk '{print $3}')
- Recreate original database-storage-wdp-couchdb-0 PVC
The database-storage-wdp-couchdb-0 PVC points to new PV created earlier by the database-storage-wdp-couchdb-0-new PVC we created and copied the data to.
Get the volume name created by the database-storage-wdp-couchdb-0-new PVC.
export PV_NAME_WDP_COUCHDB_0=$(oc get pvc database-storage-wdp-couchdb-0-new --output jsonpath={.spec.volumeName} -n ${PROJECT_CPD_INST_OPERANDS})
Create the yaml file of the new database-storage-wdp-couchdb-0 PVC.
oc get pvc database-storage-wdp-couchdb-0 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-0-recreate.json
tmp=$(mktemp)
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-0-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-0-recreate.json
Refer to the new PV.
jq --arg PV_NAME_WDP_COUCHDB_0 "$PV_NAME_WDP_COUCHDB_0" '.spec.volumeName = $PV_NAME_WDP_COUCHDB_0' pvc-database-storage-wdp-couchdb-0-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-0-recreate.json
Remove the old and new PVCs for wdp-couchdb-0
oc delete pvc database-storage-wdp-couchdb-0-new -n ${PROJECT_CPD_INST_OPERANDS}
oc delete pvc database-storage-wdp-couchdb-0 -n ${PROJECT_CPD_INST_OPERANDS}
Remove the claimRef section from the new PV.
oc patch pv $PV_NAME_WDP_COUCHDB_0 -p '{"spec":{"claimRef": null}}'
Recreate the database-storage-wdp-couchdb-0 PVC.
oc apply -f pvc-database-storage-wdp-couchdb-0-recreate.json
Make sure the new PVC is created and bound successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-0
2.3.4 Migration for the database-storage-wdp-couchdb-1 pvc
- Create a new PVC by referencing the database-storage-wdp-couchdb-1 pvc.
oc get pvc database-storage-wdp-couchdb-1 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-1-new.json
Specify a new name and the right storage class (ocs-storagecluster-ceph-rbd) for the new PVC.
tmp=$(mktemp)
jq '.metadata.name = "database-storage-wdp-couchdb-1-new"' pvc-database-storage-wdp-couchdb-1-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-1-new.json
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-1-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-1-new.json
oc apply -f pvc-database-storage-wdp-couchdb-1-new.json
Make sure the new PVC is created successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-1-new
Scale down the sleep deployment
oc scale deploy sleep --replicas=0
Wait until it's scaled down .
- Mount the old database-storage-wdp-couchdb-1 PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=old-claim --claim-name=database-storage-wdp-couchdb-1 --mount-path=/old-claim
- Mount the new database-storage-wdp-couchdb-1-new PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=new-claim --claim-name=database-storage-wdp-couchdb-1-new --mount-path=/new-claim
- Scale back the sleep deployment
oc scale deploy sleep --replicas=1
Wait until the sleep pod is up and running.
oc get pods | grep -i sleep
oc rsh $(oc get pod | grep sleep | awk '{print $1}')
- Migrate data to the new storage:
rsync -avxHAX --progress /old-claim/* /new-claim
Note: Make sure the termial session will not be closed or expired during this step.
Ensure the number of files between the old-claim and the new-claim folder is same. For example:
sh-4.2$ cd old-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
sh-4.2$ cd ../new-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
- Remove the volume mounts from the sleep deployment
oc set volume deployment sleep --remove --confirm
- Patch the PV of the database-storage-wdp-couchdb-1-new PVC for chaning the ReclaimPolicy to be "Retain"
oc patch pv $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep wdp-couchdb-1-new | awk '{print $3}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' -n ${PROJECT_CPD_INST_OPERANDS}
- Recreate original database-storage-wdp-couchdb-1 PVC
The database-storage-wdp-couchdb-1 PVC points to new PV created earlier by the database-storage-wdp-couchdb-1-new PVC we created and copied the data to.
Get the volume name created by the database-storage-wdp-couchdb-1-new PVC.
export PV_NAME_WDP_COUCHDB_1=$(oc get pvc database-storage-wdp-couchdb-1-new --output jsonpath={.spec.volumeName} -n ${PROJECT_CPD_INST_OPERANDS})
Create the yaml file of the new database-storage-wdp-couchdb-1 PVC.
oc get pvc database-storage-wdp-couchdb-1 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-1-recreate.json
tmp=$(mktemp)
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-1-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-1-recreate.json
Refer to the new PV.
jq --arg PV_NAME_WDP_COUCHDB_1 "$PV_NAME_WDP_COUCHDB_1" '.spec.volumeName = $PV_NAME_WDP_COUCHDB_1' pvc-database-storage-wdp-couchdb-1-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-1-recreate.json
Remove the old and new PVCs for wdp-couchdb-1
oc delete pvc database-storage-wdp-couchdb-1-new -n ${PROJECT_CPD_INST_OPERANDS}
oc delete pvc database-storage-wdp-couchdb-1 -n ${PROJECT_CPD_INST_OPERANDS}
Remove the claimRef section from the new PV.
oc patch pv $PV_NAME_WDP_COUCHDB_1 -p '{"spec":{"claimRef": null}}'
Recreate the database-storage-wdp-couchdb-1 PVC.
oc apply -f pvc-database-storage-wdp-couchdb-1-recreate.json
Make sure the new PVC is created and bound successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-1
2.3.5 Migration for the database-storage-wdp-couchdb-2 pvc
- Create a new PVC by referencing the database-storage-wdp-couchdb-2 pvc.
oc get pvc database-storage-wdp-couchdb-2 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-2-new.json
Specify a new name and the right storage class (ocs-storagecluster-ceph-rbd) for the new PVC.
tmp=$(mktemp)
jq '.metadata.name = "database-storage-wdp-couchdb-2-new"' pvc-database-storage-wdp-couchdb-2-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-2-new.json
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-2-new.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-2-new.json
Create the wdp-couchdb-2-new pvc.
oc apply -f pvc-database-storage-wdp-couchdb-2-new.json
Make sure the new PVC is created successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-2-new
Scale down the sleep deployment
oc scale deploy sleep --replicas=0
Wait until it's scaled down .
- Mount the old database-storage-wdp-couchdb-2 PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=old-claim --claim-name=database-storage-wdp-couchdb-2 --mount-path=/old-claim
- Mount the new database-storage-wdp-couchdb-2-new PVC to the sleep pod
oc set volume deployment/sleep --add -t pvc --name=new-claim --claim-name=database-storage-wdp-couchdb-2-new --mount-path=/new-claim
- Scale back the sleep deployment
oc scale deploy sleep --replicas=1
Wait until the sleep pod is up and running.
oc get pods | grep -i sleep
oc rsh $(oc get pod | grep sleep | awk '{print $1}')
- Migrate data to the new storage:
rsync -avxHAX --progress /old-claim/* /new-claim
Note: Make sure the termial session will not be closed or expired during this step.
Ensure the number of files between the old-claim and the new-claim folder is same. For example:
sh-4.2$ cd old-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
sh-4.2$ cd ../new-claim/
sh-4.2$ ls | while read dir; do printf "%-25.45s : " "$dir"; ls -R "$dir" | sed '/^[[:space:]]*$/d' | wc -l; done
_dbs.couch : 1
_nodes.couch : 1
search_indexes : 1749
shards : 297
- Remove the volume mounts from the sleep deployment
oc set volume deployment sleep --remove --confirm
- Patch the PV of the database-storage-wdp-couchdb-2-new PVC for chaning the ReclaimPolicy to be "Retain"
oc patch pv $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep wdp-couchdb-2-new | awk '{print $3}') -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' -n ${PROJECT_CPD_INST_OPERANDS}
- Recreate original database-storage-wdp-couchdb-2 PVC
The database-storage-wdp-couchdb-2 PVC points to new PV created earlier by the database-storage-wdp-couchdb-2-new PVC we created and copied the data to.
Get the volume name created by the database-storage-wdp-couchdb-2-new PVC.
export PV_NAME_WDP_COUCHDB_2=$(oc get pvc database-storage-wdp-couchdb-2-new --output jsonpath={.spec.volumeName} -n ${PROJECT_CPD_INST_OPERANDS})
Create the yaml file of the new database-storage-wdp-couchdb-2 PVC.
oc get pvc database-storage-wdp-couchdb-2 -o json | jq 'del(.status)'| jq 'del(.metadata.annotations)' | jq 'del(.metadata.creationTimestamp)'|jq 'del(.metadata.resourceVersion)'|jq 'del(.metadata.uid)'| jq 'del(.spec.volumeName)' > pvc-database-storage-wdp-couchdb-2-recreate.json
Change the storage class to be ocs-storagecluster-ceph-rbd.
tmp=$(mktemp)
jq '.spec.storageClassName = "ocs-storagecluster-ceph-rbd"' pvc-database-storage-wdp-couchdb-2-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-2-recreate.json
Refer to the new PV.
jq --arg PV_NAME_WDP_COUCHDB_2 "$PV_NAME_WDP_COUCHDB_2" '.spec.volumeName = $PV_NAME_WDP_COUCHDB_2' pvc-database-storage-wdp-couchdb-2-recreate.json > "$tmp" && mv -f "$tmp" pvc-database-storage-wdp-couchdb-2-recreate.json
Remove the old and new PVCs for wdp-couchdb-2
oc delete pvc database-storage-wdp-couchdb-2-new -n ${PROJECT_CPD_INST_OPERANDS}
oc delete pvc database-storage-wdp-couchdb-2 -n ${PROJECT_CPD_INST_OPERANDS}
Remove the claimRef section from the new PV.
oc patch pv $PV_NAME_WDP_COUCHDB_2 -p '{"spec":{"claimRef": null}}'
Recreate the database-storage-wdp-couchdb-2 PVC.
oc apply -f pvc-database-storage-wdp-couchdb-2-recreate.json
Make sure the new PVC is created and bound successfully.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep database-storage-wdp-couchdb-2
2.3.6 Scale the wdp-couchdb statefulset back
oc scale sts wdp-couchdb --replicas=3 -n ${PROJECT_CPD_INST_OPERANDS}
2.4.Change the ReclaimPolicy back to be "Delete" for the PVs
Patch the CouchDB PVs.
for p in $(oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} | grep couchdb | awk '{print $3}') ;do oc patch pv $p -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' -n ${PROJECT_CPD_INST_OPERANDS};done
2.5.Make sure the correct storage type is specified in CCS cr
oc patch ccs ccs-cr --type merge --patch '{"spec": {"blockStorageClass": "ocs-storagecluster-ceph-rbd", "fileStorageClass": "ocs-storagecluster-cephfs"}}' -n ${PROJECT_CPD_INST_OPERANDS}
oc get ccs ccs-cr -oyaml
2.6.Get the CCS cr out of the maintenance mode
Get the CCS cr out of the maintenance mode to trigger the operator reconcilation.
oc patch ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": false}}' -n ${PROJECT_CPD_INST_OPERANDS}
- Make sure the CCS custom resource is in 'Completed' status and also with the right storage classes.
oc get CCS cr -n ${PROJECT_CPD_INST_OPERANDS}
- Make sure all the services are in 'Completed' status.
Run the cpd-cli manage login-to-ocp command to log in to the cluster.
cpd-cli manage login-to-ocp \
--username=${OCP_USERNAME} \
--password=${OCP_PASSWORD} \
--server=${OCP_URL}
Get all services' status.
cpd-cli manage get-cr-status --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS}
- Make sure the migration relevant pods are up and running.
oc get pods -n ${PROJECT_CPD_INST_OPERANDS}| grep -E "wdp-couchdb"
- Make sure the migration relevant PVC are in 'Bound' status and also with the right storage classes.
oc get pvc -n ${PROJECT_CPD_INST_OPERANDS}| grep -E "wdp-couchdb"
- Conduct user acceptance tests
Comprehensive tests should be done by end-users. It includes accessing existing data and creating new data using multiple end-users.
oc -n ${PROJECT_CPD_INST_OPERANDS} delete deployment sleep