Writer: Jürgen Neth
Reviewer: Larissa Auberger, Benjamin Wende
Have you ever been in the situation that something get wrong inside your User Management Service (UMS) installation and you want to find the cause, but you don't know how to start?
This blog explains how to get the logs of the different UMS capabilities and the ICP4A operator and how to read the content of these.
Since version 20.0.3 UMS stores the logs of all running and terminated pods into a persistentVolume.
This shared volume for logs is defined in your operator.yaml
volumes:
- name: logs-backup
persistentVolumeClaim:
claimName: "cp4a-shared-log-pvc"
You can also specify a UMS specific log PersistentVolumeClaim in the Custom Resource file in the ums_configuration section:
ums_configuration:
existing_claim_name_logstore: <your UMS PersistentVolumeClaim>
Before taking a look into the logs, verify if the operator pod is up and running and if the UMS pods are up and running.
- Collect information about the status of all pods
oc get pod
- If the status of a pod is different to "Running" there might be a problem.
- Check the events of these pods to get more information about potential problems
oc get event --field-selector involvedObject.name=<pod>
- If you want to see more details about a pod use:
oc describe pod <pod>
- Depending on the problem check the UMS capability log, the operator log or both.
Get and analyse operator log
The ibm-cp4a-operator locates the Cloud Pak base images and has Ansible roles to handle the reconciliation logic and declare a set of playbook tasks for each component. The UMS role specifies variables and defaults for how the role is executed.
- Get Operator logs for current reconcile
To see the logs of the operator container, run the following command.
oc logs deployment/ibm-cp4a-operator > operator.log
- or get the operator logs for all reconciles and copy it to a local directory
oc cp <operator pod>:logs/<operator pod> <local directory>
- The operator logs directory structure looks like:
<local directory>/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/<namespace>/<cr metadata name>/artifacts/reconcile-<#>
- Inside the operator log grep for "FAILED:" to find failed operator tasks
Get and analyse UMS logs for one running pod
- Get logs of one running UMS pod in JSON format
oc logs <ums pod>
- To turn IBM Liberty JSON logs into human-readable format use Log-Formatter
oc logs >ums pod> | logFormat > ums.log
Get log files and ffdc directories from a volume for all pods (running and terminated)
- Create a temporary pod to access Persistent Volume and copy the UMS capability logs to a local directory
If you use a seperate volume for UMS, different to the shared volume defined in the operator.yaml you have to adapt the copy-log-pod.yaml below
apiVersion: v1
kind: Pod
metadata:
name: copy-log-pod
spec:
containers:
- name: alpine
image: alpine:latest
command: ["sleep", "infinity"]
volumeMounts:
- name: cp4a-shared-log-pvc # adapt this value if you defined an existing_claim_name_logstore
mountPath: /data
volumes:
- name: cp4a-shared-log-pvc # adapt this value if you defined an existing_claim_name_logstore
persistentVolumeClaim:
claimName: cp4a-shared-log-pvc # adapt this value if you defined an existing_claim_name_logstore
- Run the script get_UMS_log.sh to collect the UMS logs based on the persistentVolume and persistentVolumeClaim defined in the copy-log-pod.yaml
- To turn IBM Liberty JSON logs into human-readable format use Log-Formatter e.g:
cat <local target log directory>/UMS/sso/<sso pod>/liberty-message.log | logformat
- Review the log files and traces at the time of the problem to try to determine the source of the problem.
Grepping for the following keyword inside the UMS logs might lead you to the cause of the problem
grep '"loglevel":"WARNING"\|"loglevel":"SEVERE"\|"type":"liberty_ffdc"|"E:"
- Check these locations for known issues: