Cloud Pak for Data

 View Only

Cloud Pak for Data: Steps for manually renewing the Cloud Pak for Data certificates for avoiding unplanned outages.

By BHARATH DEVARAJU posted Wed July 10, 2024 12:29 AM

  

Cloud Pak for Data uses certificate manager for managing the lifecycle of internal certificates. These internal certificates are configured to be automatically renewed, for example internal-tls-certificate is renewed once every 60 days. Whenever the certificates are renewed the pods mounting the secrets are automatically restarted to facilitate the availability of new certificates for the applications. This process can affect the availability of your applications resulting in downtimes until the pods are restarted. 

In the following article we aim to demonstrate a method for manually renewing the certificates during a planned maintenance activity and avoid potential downtime for your applications. 

Pre-requisties

Install the following dependencies on your bastion node

  •  Install Golang package i.e, dnf -y install golang
  •  Install cmtl utility by executing following commands,
OS=$(go env GOOS); ARCH=$(go env GOARCH); curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_${OS}_${ARCH}
chmod +x cmctl
sudo mv cmctl /usr/local/bin

Please Note  - Once the certificates are renewed the application pods are redeployed, so please make sure your image registry is still accessible and your pull secret is still valid. Reference - https://www.ibm.com/docs/en/cloud-paks/cp-data/4.8.x?topic=information-obtaining-your-entitlement-api-key

The cert-manager command line tool (cmctl)  for managing the Cloud Pak for Data certificates

The cert-manager command line tool (cmctl) can help you in managing the cert-manager and its resources on your Cloud pak for data cluster.

To test whether the cmctl tool is setup properly run the cmctl help command as shown following

$ cmctl help

cmctl is a CLI tool manage and configure cert-manager resources for Kubernetes

Usage: cmctl [command]

Available Commands:
  approve      Approve a CertificateRequest
  check        Check cert-manager components
  completion   Generate completion scripts for the cert-manager CLI
  convert      Convert cert-manager config files between different API versions
  create       Create cert-manager resources
  deny         Deny a CertificateRequest
  experimental Interact with experimental features
  help         Help about any command
  inspect      Get details on certificate related resources
  renew        Mark a Certificate for manual renewal
  status       Get details on current status of cert-manager resources
  upgrade      Tools that assist in upgrading cert-manager
  version      Print the cert-manager CLI version and the deployed cert-manager version

Flags:
  -h, --help                           help for cmctl
      --log-flush-frequency duration   Maximum number of seconds between log flushes (default 5s)

Use "cmctl [command] --help" for more information about a command.

Before running any cmtl commands, please login to your RedHat OpenShift cluster using oc login command, and the user logged in should have access to the cloud pak for data projects.  

First review the availability of your cert-manager API by running following command. If the cert-manager API is not ready, please work with IBM support team to address the underlying issue.

cmctl check api 
The cert-manager API is ready

To review any existing certificates on your cluster, first list all the certificates that are deployed on your cluster by running following command,

oc get certificates
NAME                                           READY   SECRET                             AGE    EXPIRATION
common-web-ui-ca-cert                          True    common-web-ui-cert                 147d   2025-03-18T03:14:34Z
cs-ca-certificate                              True    cs-ca-certificate-secret           154d   2026-02-05T07:50:07Z
db2oltp-internal-tls-certificate               True    db2oltp-internal-tls               153d   2025-04-07T10:33:58Z
ibm-zen-metastore-edb-certificate              True    ibm-zen-metastore-edb-secret       154d   2024-09-03T08:51:22Z
icp-mongodb-client-cert                        True    icp-mongodb-client-cert            147d   2026-02-13T03:15:08Z
identity-provider-cert                         True    identity-provider-secret           147d   2025-03-18T03:13:56Z
internal-tls-certificate                       True    internal-tls                       154d   2024-09-03T08:46:04Z
internal-tls-pkcs12-certificate                True    internal-tls-pkcs12                154d   2024-09-03T08:46:12Z

use cmctl status command to review the details of any deployed certificates on your cluster, for example, here is the command to view the details of internal-tls-certificate

cmctl status certificate internal-tls-certificate
Name: internal-tls-certificate
Namespace: zen
Created at: 2024-02-06T00:45:48-08:00
Conditions:
  Ready: True, Reason: Ready, Message: Certificate is up to date and has not expired
DNS Names:
- *.zen
- *.zen.svc
- *.zen.svc.cluster.local
- zen-ca-cert.zen
- zen-ca-cert.zen.svc
- internal-nginx-svc
- *.svc.cluster.local
- api-svc
- *.api
- ibm-nginx-svc
- localhost
- dsxl-api
Events:  <none>
Issuer:
  Name: zen-tls-issuer
  Kind: Issuer
  Conditions:
    Ready: True, Reason: KeyPairVerified, Message: Signing CA verified
  Events:  <none>
Secret:
  Name: internal-tls
  Issuer Country: 
  Issuer Organisation: 
  Issuer Common Name: cs-ca-certificate
  Key Usage: Digital Signature, Key Encipherment
  Extended Key Usages: 
  Public Key Algorithm: RSA
  Signature Algorithm: SHA256-RSA
  Subject Key ID: 
  Authority Key ID: b3f1da0c5d1ebcf62334e1a7aae877fc22cd81e7
  Serial Number: 6d90455a7a62278aa5008ca870e5f9d7
  Events:  <none>
Not Before: 2024-06-05T01:46:04-07:00
Not After: 2024-09-03T01:46:04-07:00
Renewal Time: 2024-08-04T01:46:04-07:00
No CertificateRequest found for this Certificate

Finally, use the cmctl renew command to renew all the certificates deployed in your CPD project/namespace as shown following,

cmctl renew -n zen --all

Manually triggered issuance of Certificate zen/common-web-ui-ca-cert
Manually triggered issuance of Certificate zen/cs-ca-certificate
Manually triggered issuance of Certificate zen/db2oltp-internal-tls-certificate
Manually triggered issuance of Certificate zen/ibm-zen-metastore-edb-certificate
Manually triggered issuance of Certificate zen/icp-mongodb-client-cert
Manually triggered issuance of Certificate zen/identity-provider-cert
Manually triggered issuance of Certificate zen/internal-tls-certificate
Manually triggered issuance of Certificate zen/internal-tls-pkcs12-certificate
Manually triggered issuance of Certificate zen/internal-tls-pkcs8-certificate
Manually triggered issuance of Certificate zen/mongodb-root-ca-cert
Manually triggered issuance of Certificate zen/platform-auth-cert
Manually triggered issuance of Certificate zen/platform-identity-management
Manually triggered issuance of Certificate zen/saml-auth-cert
Manually triggered issuance of Certificate zen/zen-metastore-edb-replica-client-certificate
Manually triggered issuance of Certificate zen/zen-metastore-edb-server-certificate
Manually triggered issuance of Certificate zen/zen-minio-certificate

Please note, if your cloud pak for data cluster uses multiple namespaces/projects , please list all of them in your cmctl renew command.

Conclusion

By manually renewing the certificates during a maintenance window, we avoided an unplanned outage of our Cloud Pak for Data applications.

Additional References
  • https://cert-manager.io/docs/reference/cmctl/
1 comment
17 views

Permalink

Comments

Wed July 10, 2024 01:54 AM

Thanks for the concise content !