BPM, Workflow, and Case

How to monitor operator log during IBM Business Automation Workflow on container install

By DA GUANG SUN posted Sun December 27, 2020 11:33 PM


The IBM Business Automation workflow on container install process can be divided into four stages Generally speaking. Planning, preparing, installing and completing. This blog can help you on the installing stage.
In install stage you will have target operator installed. And the operator will help you deploy the target environment according to your inputs with custom resource and secrets. The install process may lasting hours. So it's good for your to learn some skills to help you monitor and debug the install process.
First of all. Before your install. You must read the document carefully to plan and prepare your install. https://www.ibm.com/support/knowledgecenter/SSYHZ8_20.0.x/welcome/kc_welcome_dba_distrib.html
Then in the install process. You will get an running operator pod on your environment. The operator is deployed on kubernetes with an deployment like below:

# oc get deployment ibm-cp4a-operator
ibm-cp4a-operator 1/1 1 1 10h

Usually it will contain one pod only. You can get operator log with oc command like below

# oc get pod
ibm-cp4a-operator-7c8f6477c4-qld4j 1/1 Running 0 10h

# oc logs ibm-cp4a-operator-7c8f6477c4-qld4j
2020-12-28T04:03:03.839Z INFO proxy Skipping cache lookup {"resource": {"IsResourceRequest":false,"Path":"/version","Verb":"get","APIPrefix":"","APIGroup":"","APIVersion":"","Namespace":"","Resource":"","Subresource":"","Name":"","Parts":null}}
2020-12-28T04:03:03.843Z INFO proxy Skipping cache lookup {"resource": {"IsResourceRequest":false,"Path":"/version/openshift","Verb":"get","APIPrefix":"","APIGroup":"","APIVersion":"","Namespace":"","Resource":"","Subresource":"","Name":"","Parts":null}}

Please pay attention that the operator pod name "ibm-cp4a-operator-7c8f6477c4-qld4j" changes in different environment. You must change the pod name accordingly.
Due to the output format. The logs you get from the operator may hard to be read. In order to help your easy reading. You can get the reconcile based log inside the operator pod like below

# oc rsh ibm-cp4a-operator-7c8f6477c4-qld4j
sh-4.4$ cd /tmp/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/daily-db2onprem-baa-cr-ent-aad-dp/cicd-bawd-cr/artifacts/
sh-4.4$ ls
1151812229553087684 2221029477208656602 3449410703867851244 5202458289597147449 5877998682327914191 6489824846099213448 8095790652009818071
1312807562124832281 2416761729010287161 4435035433568616300 5575372128400727406 6158972322187069841 6950222742195309727 latest
1413335359942951596 2534165218760536547 5156244947572874734 5673086971096039456 6391869277755047660 7874815402956485677

You will get several folders in that folder based on what stage the deployment is. If this is the first run of the pod. The it will contain only one folder. The latest folder will be linked to the last reconcile run folder if we have.
inside the folder you will get some folders and files

sh-4.4$ cd latest/
sh-4.4$ ls
command fact_cache job_events rc status stdout

The most importance one will be the stdout. It will contains well formatted reconcile log output which marked with different color and easy to read.

sh-4.4$ cat stdout
ansible-playbook 2.9.10
config file = /etc/ansible/ansible.cfg
configured module search path = ['/usr/share/ansible/openshift']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 3.6.8 (default, Aug 18 2020, 08:33:21) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: a2e7cf3b780c4faf947f236690aa635e *************************************
1 plays in /tmp/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/daily-db2onprem-baa-cr-ent-aad-dp/cicd-bawd-cr/project/a2e7cf3b780c4faf947f236690aa635e

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
Monday 28 December 2020 03:55:54 +0000 (0:00:00.094) 0:00:00.094 *******
ok: [localhost]
META: ran handlers

TASK [icp4a : ICP4A Deploy] ****************************************************
task path: /opt/ansible/roles/icp4a/tasks/main.yml:13
Monday 28 December 2020 03:55:55 +0000 (0:00:01.023) 0:00:01.117 *******
ok: [localhost] => {
"msg": "Start ICP4A Deploy"

TASK [icp4a : License Check debug] *********************************************
task path: /opt/ansible/roles/icp4a/tasks/main.yml:18
Monday 28 December 2020 03:55:55 +0000 (0:00:00.024) 0:00:01.142 *******
ok: [localhost] => {
"msg": "ibm_license is 'accept', the valid value is 'accept' "

You can get useful errors in this log file during install. For example. If you haven't provide required admin secret fields. You will get error message in this log.

logs in that folder will be rollout if the reconcile processed more than a limits. But usually the most importance log is the first reconcile log. So that we managed a log backup in the operator logs folder to backup the first 10 reconcile logs and the latest one

sh-4.4$ cd /logs/<operator-pod-name>/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/daily-db2onprem-baa-cr-ent-aad-dp/cicd-bawd-cr/artifacts/
sh-4.4$ ls
latest reconcile-0 reconcile-1 reconcile-2 reconcile-3 reconcile-4 reconcile-5 reconcile-6 reconcile-7 reconcile-8 reconcile-9

If you meet any error during install and want to engage technical support. You'd better to collect all the files in this folder to help us determine if the operator meet any error during install. Usually the fodler has been mount to an external persistence volume. So that you can collect from the PV if you know how to.
Otherwise. You can collect that with oc command also

#oc cp ibm-cp4a-operator-7c8f6477c4-qld4j:/logs/ibm-cp4a-operator-7c8f6477c4-qld4j/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/daily-db2onprem-baa-cr-ent-aad-dp/cicd-bawd-cr/artifacts ./
tar: Removing leading `/' from member names
# ls
latest reconcile-0 reconcile-1 reconcile-2 reconcile-3 reconcile-4 reconcile-5 reconcile-6 reconcile-7 reconcile-8 reconcile-9
# cd reconcile-0
# ls


1 comment



Thu January 07, 2021 04:21 AM

Exactly what I was looking for. Thank you for sharing this with us