In certain customer scenarios, clients may be mandated to uninstall their existing Cloud Pak for Business Automation (CP4BA) deployment prior to reinstalling it. This requirement could stem from various reasons, including installation failure, critical component malfunction, non-compliance with prerequisites, mid-installation requirement changes, high resource consumption affecting existing workloads, cluster permissions or security policy restrictions, or version incompatibility. During the uninstallation process, users frequently face challenges such as incomplete uninstallation or uninstallation failures, which can subsequently result in another failure when attempting to reinstall.
This blog post will take CP4BA 24.0.1with Cloud Pak Foundation Services (CPFS) 4.11 as an example, and concentrates on seven prevalent uninstallation scenarios, aiming to assist users in comprehending the resources that necessitate cleanup during the process and how to execute this cleanup effectively. By ensuring a thorough uninstallation, users can pave the way for a seamless reinstallation post-cleanup.
Scenario 1, Uninstalling CP4BA: Operators and Operands in the Same Namespace
In this scenario, your OpenShift Container Platform (OCP) cluster hosts only the CP4BA deployment, with operators and operands residing within the same namespace. For this example, let's assume that your CP4BA is installed in the "cp4ba" namespace.
1, remove CP4BA Custom Resource (CR)instance:
oc delete icp4acluster –all -n cp4ba
2, Allow some time for all pods to be evicted, ensuring no pods remain in the "terminating" status. If you observe one or more pods persisting in the "terminating" state for an extended period, you may manually intervene to forcibly delete them.
oc delete pod <podname> --force –grace-period=0 -n cp4ba
3, Upon the removal of the CP4BA CR instance, all resources deployed by CP4BA will be automatically eliminated. Nevertheless, certain resources may not be purged, necessitating manual intervention from the end user to eliminate finalizers, such as Clients, Zenextension, and FlinkDeployment (if Business Automation Insights was installed). To accomplish this, the end user can execute the subsequent commands to patch the finalizers, subsequently facilitating the removal of the resources:
for i in $(oc -n cp4ba get zenextension --no-headers | awk '{print $1}'); do
oc -n cp4ba patch zenextension/$i -p '{"metadata":{"finalizers":[]}}' --type=merge
oc -n cp4ba delete zenextension $i --ignore-not-found=true --wait=true
done
for i in $(oc -n cp4ba get flinkdeployments.flink.ibm.com --no-headers| awk '{print $1}');do
oc -n cp4ba patch flinkdeployments.flink.ibm.com/$i -p '{"metadata":{"finalizers":[]}}' --type=merge
oc -n cp4ba delete flinkdeployments.flink.ibm.com $i --ignore-not-found=true --wait=true
done
for i in $(oc -n cp4ba get clients.oidc.security.ibm.com --no-headers | awk '{print $1}'); do
oc -n cp4ba patch clients.oidc.security.ibm.com/$i -p '{"metadata":{"finalizers":[]}}' --type=merge
oc -n cp4ba delete clients.oidc.security.ibm.com $i --ignore-not-found=true --wait=true
done
4, Delete CP4BA, ibm-licensing and ibm-cert-manager (optional) namespace
oc delete project cp4ba
oc delete project ibm-licensing
oc delete project ibm-cert-manager (only required you use IBM cert manager, you can skip this step if you use Red Hat Cert Manager)
5, Miscellaneous cleanup
1.5.1 Remove ConfigMaps from the “kube-public” namespace
oc delete cm common-service-maps ibm-common-services-status -n kube-public --ignore-not-found
1.5.2 Remove apiservice deployed by CP4BA and CPFS
oc delete apiservice v1.decisions.ibm.com
oc delete apiservice v1.dpe.ibm.com
oc delete apiservice v1.elasticsearch.opencontent.ibm.com
oc delete apiservice v1.foundation.ibm.com
oc delete apiservice v1.icp4a.ibm.com
oc delete apiservice v1.oidc.security.ibm.com
oc delete apiservice v1.operator.ibm.com
oc delete apiservice v1.zen.cpd.ibm.com
oc delete apiservice v1alpha1.ibmevents.ibm.com
oc delete apiservice v1alpha1.operator.ibm.com
oc delete apiservice v1alpha1.operators.ibm.com
oc delete apiservice v1beta1.ibmevents.ibm.com
oc delete apiservice v1beta2.core.ibmevents.ibm.com
oc delete apiservice v1beta2.ibmevents.ibm.com
oc delete apiservice v3.operator.ibm.com
1.5.3 Remove ValidatingWebhookConfiguration and MutatingWebhookConfiguration from CPFS
oc delete ValidatingWebhookConfiguration cert-manager-webhook
oc delete ValidatingWebhookConfiguration ibm-cs-ns-mapping-webhook-configuration
oc delete ValidatingWebhookConfiguration ibm-common-service-validating-webhook
oc delete MutatingWebhookConfiguration cert-manager-webhook
oc delete MutatingWebhookConfiguration namespace-admission-config
oc delete MutatingWebhookConfiguration ibm-operandrequest-webhook-configuration
oc delete MutatingWebhookConfiguration ibm-common-service-webhook-configuration
1.5.4 Remove ClusterRole deployed by CP4BA and CPFS
for i in $(oc get clusterrole|grep -E "postgresql.k8s.enterprisedb|ibm-cp4a|ibm-bts|ibm-ads|ibm-iam|cloudpak-|iam-|icp-|icp:"|awk '{print $1}');do oc delete clusterrole "$i";done
1.5.5 Remove ClusterRoleBinding deployed by CP4BA and CPFS
for i in $(oc get clusterrolebinding|grep -E " postgresql.k8s.enterprisedb |flink-kubernetes-operator|secretshare-cs-control|ibm-licens|ibm-automation|ibm-bts|ibm-cert-manager|ibm-common-service|ibm-cp4a-ibm-crossplane|ibm-elastic|ibm-ingress|ibm-management|ibm-namespace|ibm-platform|ibm-zen|iam-|cloudpak-|oidc-"|awk '{print $1}');do oc delete clusterrolebinding "$i";done
1.5.6 Remove RoleBinding deployed by CP4BA and CPFS from “kube-system” namespace
for i in $(oc get rolebinding -n kube-system|grep -E "ibm-bts-operator|ibm-common-service-|ibm-elastic-operator-|ibm-cp4a-|iaf-|postgresql-|flink-kubernetes-operator"|awk '{print $1}');do oc delete rolebinding "$i" -n kube-system;done
1.5.7 If you've applied a global catalog source and no longer require the reinstallation of CP4BA, consider removing the catalog source for a comprehensive cleanup (optional).
oc delete catalogsource ibm-cp4a-operator-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-fncm-operator-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource cloud-native-postgresql-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-cert-manager-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-licensing-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-cs-opensearch-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-events-operator-catalog -n openshift-marketplace --ignore-not-found=true --wait=true
oc delete catalogsource ibm-opencontent-flink -n openshift-marketplace --ignore-not-found=true --wait=true
for i in $(oc get catalogsource -n openshift-marketplace|grep ibm-iam-operator-catalog|awk '{print $1}'); do oc delete catalogsource "$i" -n openshift-marketplace;done
for i in $(oc get catalogsource -n openshift-marketplace|grep ibm-zen-operator-catalog|awk '{print $1}'); do oc delete catalogsource "$i" -n openshift-marketplace;done
for i in $(oc get catalogsource -n openshift-marketplace|grep ibm-cs-install-catalog|awk '{print $1}'); do oc delete catalogsource "$i" -n openshift-marketplace;done
for i in $(oc get catalogsource -n openshift-marketplace|grep bts-operator|awk '{print $1}'); do oc delete catalogsource "$i" -n openshift-marketplace;done
1.5.8 If you're no longer planning to reinstall CP4BA and require a comprehensive cleanup, consider deleting the CustomResourceDefinition (CRD) deployed by both CP4BA and CPFS (optional)
oc delete crd authentications.operator.ibm.com
oc delete crd backups.postgresql.k8s.enterprisedb.io
oc delete crd businessautomationmachinelearnings.icp4a.ibm.com
oc delete crd businessteamsservices.operator.ibm.com
oc delete crd certificaterequests.cert-manager.io
oc delete crd certificates.cert-manager.io
oc delete crd certmanagerconfigs.operator.ibm.com
oc delete crd challenges.acme.cert-manager.io
oc delete crd clients.oidc.security.ibm.com
oc delete crd clusterissuers.cert-manager.io
oc delete crd clusters.postgresql.k8s.enterprisedb.io
oc delete crd commonservices.operator.ibm.com
oc delete crd commonwebuis.operators.ibm.com
oc delete crd contentrequests.icp4a.ibm.com
oc delete crd contents.icp4a.ibm.com
oc delete crd documentprocessingengines.dpe.ibm.com
oc delete crd elasticsearchclusters.elasticsearch.opencontent.ibm.com
oc delete crd federatedsystems.icp4a.ibm.com
oc delete crd foundationrequests.icp4a.ibm.com
oc delete crd foundations.icp4a.ibm.com
oc delete crd ibmlicensingdefinitions.operator.ibm.com
oc delete crd ibmlicensingmetadatas.operator.ibm.com
oc delete crd ibmlicensingquerysources.operator.ibm.com
oc delete crd ibmlicensings.operator.ibm.com
oc delete crd icp4aautomationdecisionservices.icp4a.ibm.com
oc delete crd icp4aclusters.icp4a.ibm.com
oc delete crd icp4adocumentprocessingengines.icp4a.ibm.com
oc delete crd icp4aoperationaldecisionmanagers.icp4a.ibm.com
oc delete crd insightsenginerequests.icp4a.ibm.com
oc delete crd insightsengines.icp4a.ibm.com
oc delete crd issuers.cert-manager.io
oc delete crd kafkabridges.ibmevents.ibm.com
oc delete crd kafkaconnectors.ibmevents.ibm.com
oc delete crd kafkaconnects.ibmevents.ibm.com
oc delete crd kafkamirrormaker2s.ibmevents.ibm.com
oc delete crd kafkamirrormakers.ibmevents.ibm.com
oc delete crd kafkanodepools.ibmevents.ibm.com
oc delete crd kafkarebalances.ibmevents.ibm.com
oc delete crd kafkas.ibmevents.ibm.com
oc delete crd kafkatopics.ibmevents.ibm.com
oc delete crd kafkausers.ibmevents.ibm.com
oc delete crd navconfigurations.foundation.ibm.com
oc delete crd operandbindinfos.operator.ibm.com
oc delete crd operandconfigs.operator.ibm.com
oc delete crd operandregistries.operator.ibm.com
oc delete crd operandrequests.operator.ibm.com
oc delete crd operationaldecisionmanagers.decisions.ibm.com
oc delete crd operatorconfigs.operator.ibm.com
oc delete crd orders.acme.cert-manager.io
oc delete crd poolers.postgresql.k8s.enterprisedb.io
oc delete crd processfederationservers.icp4a.ibm.com
oc delete crd scheduledbackups.postgresql.k8s.enterprisedb.io
oc delete crd strimzipodsets.core.ibmevents.ibm.com
oc delete crd switcheritems.operators.ibm.com
oc delete crd wfpsruntimes.icp4a.ibm.com
oc delete crd workflowruntimes.icp4a.ibm.com
oc delete crd zenextensions.zen.cpd.ibm.com
oc delete crd zenservices.zen.cpd.ibm.com
Scenario 2, Separation of Duties Uninstallation: CP4BA with Operators and Operands in Separate Namespaces
In this scenario, your OCP cluster hosts only the CP4BA deployment, with operators and operands residing in separate namespaces. For this example, let's assume that your CP4BA operators are installed in the "cp4ba-operators" namespace, while your CP4BA operand is installed in the "cp4ba" namespace. The key difference from Scenario 1 is the removal of the NamespaceScope (nss) instance from the CP4BA operator namespace, followed by the deletion of the operandrequest (opreq) instance from the CP4BA operator namespace, and finally the deletion of the CP4BA operator namespace, as shown in the command below
oc delete project cp4ba
oc delete nss –all -n cp4ba-operators
oc delete opreq –all -n cp4ba-operators
oc delete project cp4ba-operators
oc delete project ibm-licensing
…
Scenario 3, Selective Cleanup: Uninstalling CP4BA from a Single Namespace
In this scenario, your OCP cluster may host multiple instances of CP4BA in various namespaces, or you might have multiple Cloud Paks, including CP4BA, installed in separate namespaces. However, only one CP4BA instance will be uninstalled. For this example, let's assume that your CP4BA is installed in the "cp4ba" namespace.
The initial three steps align with Scenario 1, but starting from step 4, only the CP4BA namespace should be deleted using the following command:
oc delete project cp4ba
Subsequently, all the remaining steps from Scenario 1 can be skipped, and the end user should refrain from performing any additional Miscellaneous cleanup.
Scenario 4, Uninstalling CP4BA Alongside Workflow Process Service Runtime (WFPS Runtime)
In this scenario, your OCP cluster hosts only the CP4BA deployment, with operators and operands residing within the same namespaces. Additionally, you have the WFPS Runtime operand in the same namespace. For this uninstallation, the steps closely resemble Scenario 1, with an extra step required after removing the CP4BA CR instance. This additional step involves eliminating the WFPS Runtime CR instance using the following command:
cc delete wfpsruntime –all -n cp4ba
The remaining steps align with Scenario 1.
Scenario 5, Comprehensive Uninstallation: CP4BA in an All-Namespace Deployment
In this scenario, your OCP cluster hosts only the CP4BA deployment, with operators and operands residing in separate namespaces. For this example, let's assume that your CP4BA operand is installed in the "cp4ba" namespace, while your operators are situated in the "openshift-operators" namespace. The uninstallation process deviates from Scenario 1, as CPFS will be installed in the "ibm-common-services" namespace, and all catalog sources must be global.
The initial four steps align with Scenario 1, but in step 5 (Miscellaneous cleanup), the process diverges. Prior to eliminating the catalog source and all Custom Resource Definitions (CRDs), the end user should also delete the "ibm-common-services" namespace.
oc delete project “ibm-common-services” namespace
Then “openshift-operators” namespace should also be cleaned up
5.1 Remove all operandrequests from openshift-operators namespace
oc -n openshift-operators delete operandrequest --force --grace-period=0 --all --ignore-not-found=true --wait=true
5.2, Remove all subscrptions and ClusterServiceVersion (CSV) created by CP4BA and CPFS from openshift-operators namespace
oc delete csv,sub -n openshift-operators --all --ignore-not-found=true --wait=true
5.3, Remove all configmaps created by CP4BA and CPFS from openshift-operators namespace
oc -n openshift-operators get cm | grep -E "ibm|namespace-scope" | awk '{print $1}' | xargs oc delete cm -n openshift-operators --ignore-not-found=true
5.4, Remove all service accounts created by CP4BA and CPFS from openshift-operators namespace
oc -n openshift-operators get sa | grep -E "ibm|postgresql" | awk '{print $1}' | xargs oc delete sa -n openshift-operators --ignore-not-found=true
5.5, Remove lease, secret, service, networkpolicy, job, deployment, role, and Persistent Volumes
oc delete lease,secret,svc,netpol,job,deploy,pvc,role --all -n openshift-operators --ignore-not-found=true
5.6, Remove CPFS related CRs from openshift-operators namespace
oc delete commonservice,operandregistry,operandconfig --all -n openshift-operators --ignore-not-found=true --wait=true
Then the end user could continue to follow the steps 5.7 and 5.8 in Scenario 1
Scenario 6, Uninstalling FileNet Content Manager (FNCM) installed from the Content Operator
In this scenario, your OCP cluster hosts only the FileNet Content Manager (FNCM) deployment from Content CR, with operators and operands residing within the same namespace. For this example, let's assume that your Content is installed in the "fncm" namespace. For this uninstallation, the steps closely resemble Scenario 1, with the primary distinction being that only the Content CR instance will be removed, as demonstrated by the following command:
oc delete content –all -n fncm
The remaining steps align with Scenario 1.
Scenario 7, Uninstalling CP4BA with IBM Process Mining (IPM) and/or IBM Robotic Process Automation (RPA)
In this scenario, your OCP cluster accommodates CP4BA IPM and/or RPA. For this example, let's assume that your CP4BA, IPM, and RPA are all installed within the "cp4ba" namespace. As a best practice, IPM and/or RPA should be uninstalled prior to CP4BA.
7.1, Remove RPA CR instance
oc delete roboticprocessautomation -n cp4ba --all
7.2, Remove IPM CR instance
oc delete processmining -n cp4ba --all
Then follow all the steps mentioned in Scenario 1.
7.3, After all namespaces are deleted and all resources are all removed, ibm-operator-catalog catalog source could also be removed from openshift-marketplace (optional)
oc delete catsrc ibm-operator-catalog -n openshift-marketplace
7.4, These CRDs can also be removed (optional)
oc delete crd backups.redis.databases.cloud.ibm.com
oc delete crd bigsqls.db2u.databases.ibm.com
oc delete crd buckets.redis.databases.cloud.ibm.com
oc delete crd commonservices.operator.ibm.com
oc delete crd db2ubackups.db2ubnr.databases.ibm.com
oc delete crd db2uclusters.db2u.databases.ibm.com
oc delete crd db2uengines.db2u.databases.ibm.com
oc delete crd db2uhadrs.db2u.databases.ibm.com
oc delete crd db2uinstances.db2u.databases.ibm.com
oc delete crd db2ulogstreams.db2ulog.databases.ibm.com
oc delete crd db2urestores.db2ubnr.databases.ibm.com
oc delete crd formationlocks.db2u.databases.ibm.com
oc delete crd formationlocks.redis.databases.cloud.ibm.com
oc delete crd formations.db2u.databases.ibm.com
oc delete crd formations.redis.databases.cloud.ibm.com
oc delete crd namespacescopes.operator.ibm.com
oc delete crd processminings.processmining.ibm.com
oc delete crd queuemanagers.mq.ibm.com
oc delete crd recipes.redis.databases.cloud.ibm.com
oc delete crd recipetemplates.redis.databases.cloud.ibm.com
oc delete crd rediscps.redis.ibm.com
oc delete crd redissentinels.redis.databases.cloud.ibm.com
oc delete crd releases.redis.databases.cloud.ibm.com
oc delete crd roboticprocessautomations.rpa.automation.ibm.com
For all the mentioned scenarios, when deleting the CP4BA namespace, it might become stuck due to finalizers on certain resources. In such cases, you can execute the following command to identify the hanging resources:
oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n <namespace> 2>/dev/null | grep -v "Warning:" | grep -E -v "packagemanifest|Normal|Warning|NAME|MESSAGE|message:"
In the provided example, after deleting the "production" namespace, it remained in the "Terminating" status for an extended period. The end user then discovered the hanging resource, which was "client.oidc.security.ibm.com/zenclient-production". Subsequently, the end user ran the command to eliminate the finalizers, successfully deleting the "production" namespace.
For all the previously mentioned scenarios, if you utilize an external database, remember to recreate all your databases for the reinstallation of CP4BA.
Conclusion:
In these seven uninstallation scenarios, we have detailed the process of uninstalling CP4BA in various namespace and configuration setups. Each scenario provides specific steps and commands to ensure users can thoroughly clean up all related resources and eliminate any potential obstacles that could cause errors during reinstallation. By following these steps, customers can guarantee a clean uninstallation of CP4BA and provide a stable and error-free foundation for reinstallation.
Reference:
https://www.ibm.com/docs/en/cloud-paks/cp-biz-automation/24.0.1?topic=deployment-uninstalling-capabilities
https://www.ibm.com/docs/en/rpa/23.0?topic=server-uninstalling-rpa-operator