Cloud Pak for Data uses certificate manager for managing the lifecycle of internal certificates. These internal certificates are configured to be automatically renewed, for example internal-tls-certificate is renewed once every 60 days. Whenever the certificates are renewed the pods mounting the secrets are automatically restarted to facilitate the availability of new certificates for the applications. This process can affect the availability of your applications resulting in downtimes until the pods are restarted.
In the following article we aim to demonstrate a method for manually renewing the certificates during a planned maintenance activity and avoid potential downtime for your applications.
Pre-requisties
Install the following dependencies on your bastion node
- Install Golang package i.e, dnf -y install golang
- Install cmtl utility by executing following commands,
OS=$(go env GOOS); ARCH=$(go env GOARCH); curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_${OS}_${ARCH}
chmod +x cmctl
sudo mv cmctl /usr/local/bin
Please Note - Once the certificates are renewed the application pods are redeployed, so please make sure your image registry is still accessible and your pull secret is still valid. Reference - https://www.ibm.com/docs/en/cloud-paks/cp-data/4.8.x?topic=information-obtaining-your-entitlement-api-key
The cert-manager command line tool (cmctl) for managing the Cloud Pak for Data certificates
The cert-manager command line tool (cmctl) can help you in managing the cert-manager and its resources on your Cloud pak for data cluster.
To test whether the cmctl tool is setup properly run the cmctl help command as shown following
$ cmctl help
cmctl is a CLI tool manage and configure cert-manager resources for Kubernetes
Usage: cmctl [command]
Available Commands:
approve Approve a CertificateRequest
check Check cert-manager components
completion Generate completion scripts for the cert-manager CLI
convert Convert cert-manager config files between different API versions
create Create cert-manager resources
deny Deny a CertificateRequest
experimental Interact with experimental features
help Help about any command
inspect Get details on certificate related resources
renew Mark a Certificate for manual renewal
status Get details on current status of cert-manager resources
upgrade Tools that assist in upgrading cert-manager
version Print the cert-manager CLI version and the deployed cert-manager version
Flags:
-h, --help help for cmctl
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Use "cmctl [command] --help" for more information about a command.
Before running any cmtl commands, please login to your RedHat OpenShift cluster using oc login command, and the user logged in should have access to the cloud pak for data projects.
First review the availability of your cert-manager API by running following command. If the cert-manager API is not ready, please work with IBM support team to address the underlying issue.
cmctl check api
The cert-manager API is ready
To review any existing certificates on your cluster, first list all the certificates that are deployed on your cluster by running following command,
oc get certificates
NAME READY SECRET AGE EXPIRATION
common-web-ui-ca-cert True common-web-ui-cert 147d 2025-03-18T03:14:34Z
cs-ca-certificate True cs-ca-certificate-secret 154d 2026-02-05T07:50:07Z
db2oltp-internal-tls-certificate True db2oltp-internal-tls 153d 2025-04-07T10:33:58Z
ibm-zen-metastore-edb-certificate True ibm-zen-metastore-edb-secret 154d 2024-09-03T08:51:22Z
icp-mongodb-client-cert True icp-mongodb-client-cert 147d 2026-02-13T03:15:08Z
identity-provider-cert True identity-provider-secret 147d 2025-03-18T03:13:56Z
internal-tls-certificate True internal-tls 154d 2024-09-03T08:46:04Z
internal-tls-pkcs12-certificate True internal-tls-pkcs12 154d 2024-09-03T08:46:12Z
use cmctl status command to review the details of any deployed certificates on your cluster, for example, here is the command to view the details of internal-tls-certificate
cmctl status certificate internal-tls-certificate
Name: internal-tls-certificate
Namespace: zen
Created at: 2024-02-06T00:45:48-08:00
Conditions:
Ready: True, Reason: Ready, Message: Certificate is up to date and has not expired
DNS Names:
- *.zen
- *.zen.svc
- *.zen.svc.cluster.local
- zen-ca-cert.zen
- zen-ca-cert.zen.svc
- internal-nginx-svc
- *.svc.cluster.local
- api-svc
- *.api
- ibm-nginx-svc
- localhost
- dsxl-api
Events: <none>
Issuer:
Name: zen-tls-issuer
Kind: Issuer
Conditions:
Ready: True, Reason: KeyPairVerified, Message: Signing CA verified
Events: <none>
Secret:
Name: internal-tls
Issuer Country:
Issuer Organisation:
Issuer Common Name: cs-ca-certificate
Key Usage: Digital Signature, Key Encipherment
Extended Key Usages:
Public Key Algorithm: RSA
Signature Algorithm: SHA256-RSA
Subject Key ID:
Authority Key ID: b3f1da0c5d1ebcf62334e1a7aae877fc22cd81e7
Serial Number: 6d90455a7a62278aa5008ca870e5f9d7
Events: <none>
Not Before: 2024-06-05T01:46:04-07:00
Not After: 2024-09-03T01:46:04-07:00
Renewal Time: 2024-08-04T01:46:04-07:00
No CertificateRequest found for this Certificate
Finally, use the cmctl renew command to renew all the certificates deployed in your CPD project/namespace as shown following,
cmctl renew -n zen --all
Manually triggered issuance of Certificate zen/common-web-ui-ca-cert
Manually triggered issuance of Certificate zen/cs-ca-certificate
Manually triggered issuance of Certificate zen/db2oltp-internal-tls-certificate
Manually triggered issuance of Certificate zen/ibm-zen-metastore-edb-certificate
Manually triggered issuance of Certificate zen/icp-mongodb-client-cert
Manually triggered issuance of Certificate zen/identity-provider-cert
Manually triggered issuance of Certificate zen/internal-tls-certificate
Manually triggered issuance of Certificate zen/internal-tls-pkcs12-certificate
Manually triggered issuance of Certificate zen/internal-tls-pkcs8-certificate
Manually triggered issuance of Certificate zen/mongodb-root-ca-cert
Manually triggered issuance of Certificate zen/platform-auth-cert
Manually triggered issuance of Certificate zen/platform-identity-management
Manually triggered issuance of Certificate zen/saml-auth-cert
Manually triggered issuance of Certificate zen/zen-metastore-edb-replica-client-certificate
Manually triggered issuance of Certificate zen/zen-metastore-edb-server-certificate
Manually triggered issuance of Certificate zen/zen-minio-certificate
Please note, if your cloud pak for data cluster uses multiple namespaces/projects , please list all of them in your cmctl renew command.
Conclusion
By manually renewing the certificates during a maintenance window, we avoided an unplanned outage of our Cloud Pak for Data applications.
Additional References
- https://cert-manager.io/docs/reference/cmctl/