The following blog post is about log collection for the ICP4BA 21.0.1 Operator and User Management Service (UMS).
If you find this information useful, then please hit the Recommend button above. Thank you!
Log Volumes
Starting with version 20.0.3, the User Management Service (UMS) will create log files on a persistent volume.
By default UMS will use the shared log volume named cp4a-shared-log-pvc for log files. This can also be customized using the Custom Resource property ums_configuration.existing_claim_name_logstore and point it to a Persistent Volume Claim (PVC).
Inside the UMS pods this volume is mounted to the path /logs/application/UMS (v20.0.3) or starting with v21.0.1 to the path /logs/UMS.
The directory UMS will then have a single directory for each UMS pod, as well as the operator itself, if the PVC is shared with the operator.
Example (ICP4BA v21.0.1)
/logs
|- /UMS
|- /ibm-cp4a-operator-76bc66c6ff-dtlv9
|- /ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/<namespace>/<cr-name>/artifacts/latest/
|- stdout <-- Operator Log
|- /ums-db2-ums-profiles-deployment-866dd5875c-tnm2f
|- /ums-db2-ums-scim-deployment-65455d8fc5-f5zvl
|- /ums-db2-ums-sso-deployment-6b67969b8c-mtwf9
|- /ums-db2-ums-teams-deployment-7f7c8755f9-4pxms <-- Liberty Logs
|- /ffdc
|- trace.log
|- liberty-message.log
In the example above we can see that the operator logs are placed inside a structure of sub-directories that start with the operator pod name.
Note that there are 10 historical files for earlier reconcile phases. Important to know is that there will only be a log file if at least 1 reconcile was completely processed.
The log files are named stdout inside the respective directories.
For UMS we can see a sub-directory for each pod that directly contains the IBM Liberty Server logs.
Collecting Logs
The easiest way to collect logs is via the oc command line client. The oc client features a copy command, that can also recursively copy complete directory structures. Note that the same command is also available in kubectl.
Example:
oc cp ums-db2-ums-teams-deployment-7f7c8755f9-4pxms:/logs/UMS ~/logs/UMS
The command above will copy all the log files /logs/UMS directory to the local file system. Remember to use /logs/application/UMS in v20.0.3.
The syntax here is oc cp <pod-name>:/path/in/container <target-path>
As in most cases the persistent volume contains the logs for all the pods, it is sufficient to execute this for one pod and copy all the pods logs in one step.
Compared to oc logs
One might ask, why not using the oc logs <pod> command to get the pod logs.
This has some disadvantages. With this command, you can only get logs from one pod at a time. It is kind of cumbersome and time consuming to type this for all the pods you might have.
Additionally, oc logs will output JSON formatted logs, that are not easily readable.
In contrast to that, the oc cp approach will copy all pod logs at once and have them formatted in a human-readable fashion.
Summary
With ICP4BA v20.0.3 there is a log volume (cp4a-shared-log-pvc) used to store UMS and Operator Logs. We have see how these logs can be easily collected using the oc cp command.
Reference
IBM Cloud Pak for Automation 20.0.x Knowledge Center - Prepare the Log Storage
https://www.ibm.com/support/knowledgecenter/en/SSYHZ8_20.0.x/com.ibm.dba.install/op_topics/tsk_prep_operator_storage.html
#icp4ba