Monitoring BAW in CP4BA using Health Center
The IBM Java runtime ships with an integrated diagnostic tool for monitoring the status of a running JVM:
Health Center.
This article shows how to leverage this tool for monitoring components of CP4BA, in this case Business Automation Workflow (BAW) running on Redhat OpenShift 4.8.2
Health Center provides insight on:
- CPU usage, I/O, Network utilization
- Memory usage (Heap, native memory, garbage collection)
- Threads and methods
- Class loading
You can monitor a running system
online (not recommended for production systems) or run in
headless mode, which writes information to files which can be collected and used for analysis later. You can even
attach to an already running JVM and start monitoring immediately.
This example shows BAW as part of CP4BA 21.0.3, but the steps would be similar for other releases or other Cloud Pak components running in Java (e.g. CPE)
The instructions show Health Center client being installed on Microsoft Windows but should work similarly on macOS or Linux.
Installing the Health Center client
The Health Center client is provided on the eclipse marketplace and can be installed into an existing eclipse client (also see
https://www.ibm.com/docs/en/mon-diag-tools?topic=center-installing-health-client).
NOTE: If you want to use socket mode to connect to the monitoring agent, the Health Center client must be installed on a machine that can reach the Openshift Cluster API server (see OpenShift CLI documentation).The following steps show how to install a fresh eclipse client and add the Health Center component on Microsoft Windows.
- Determine the latest supported Eclipse version:
Check supported versions on the Eclipse Marketplace under "Additional Details > Eclipse Versions". At the time of writing this article, the latest one is Eclipse 2021-03 (4.19)
- Download the "Eclipse IDE for Java Developers", unpack to a local directory
- Launch eclipse client and create a workspace
- From the Welcome Page "Launch the Eclipse Marketplace", enter "Health Center" in the Find input field
- On the "IBM Monitoring and Diagnostic Tools - Health Center" item, press the "Install" button.
- Review and Accept Licenses and restart eclipse when prompted
- After restart, select "Window > Perspective > Open Perspective > Other..." and choose any of the available Health Center perspectives:
- Health Center is now ready to be connected to an instrumented JVM or to analyze files collected in headless mode
Installing a monitoring agent
NOTE: The following examples assume the usage of the "icp4adeploy" namespace and a single instance of BAW "instance1" with one replica "server-0". Modify the code according to your environment if necessary.There are different options for instrumenting the JVM with the monitoring agent. I will show two examples:
- Socket mode (not recommended for production systems): Opens a port in the JVM so that the client can directly connect to it.
- Headless mode: Writes monitoring data to files which can be collected later on.
Instrument the JVM for "Socket Mode":
Step 1: Modify jvm.options
In order to enable the Health Center agent in socket mode, the jvm.options of the BAW component need to be amended.
The following options need to be added at a minimum.
-Xhealthcenter |
Enables the included Health Center agent, which will start listening on default port 1972 |
-Djava.rmi.server.hostname=127.0.0.1 |
Makes the RMI stubs refer to hostname 127.0.0.1, so that we can use port-forwarding with the oc client (see below) |
-Dcom.ibm.java.diagnostics.healthcenter.agent.transport=jrmp |
Changes the protocol to JRMP so we don't need to forward an extra ORB port |
NOTE: This default configuration runs without any security enabled (see https://www.ibm.com/docs/en/mon-diag-tools?topic=application-securing-health-center).
In order to add system properties and options to the JVM for BAW, you can modify the custom resource ICP4ACluster custom resource object.
One way of doing this is the oc command line:
- Edit the ICP4ACluster object:
oc edit ICP4ACluster icp4adeploy
- Find the
baw_configuration.jvm_customize_options section. If it doesn't exist, create it. Add the options as following:
jvm_customize_options: |-
-Xhealthcenter
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.ibm.java.diagnostics.healthcenter.agent.transport=jrmp
- Scale the operator down and up to shorten the reconciliation cycle:
oc scale deployment ibm-cp4a-operator --replicas=0
oc scale deployment ibm-cp4a-operator --replicas=1
- Wait for the change to propagate to the config map:
watch -n20 -g "oc get configmap icp4adeploy-instance1-baw-server-configmap-liberty -o yaml | grep -C1 healthcenter"
- Restart the BAW pod:
oc scale statefulset icp4adeploy-instance1-baw-server --replicas=0
oc scale statefulset icp4adeploy-instance1-baw-server --replicas=1
Step 2: Make the agent port available on the Health Center client machine and connect:
The easiest way to access the port opened in the monitored pod is to forward that port temporarily to the client machine using the oc command line tools. This does not require changes to the clusters network configuration.
- Download the OpenShift command line tool, e.g. from the OpenShift graphical console (for more options and details, see OpenShift CLI documentation)
- Copy the login command from the console and paste it into the command line on your client machine:
- Connect the oc command line tool to the OpenShift cluster and initiate a port forwarding, e.g.:
oc port-forward icp4adeploy-instance1-baw-server-0 1972
- In Health Center client, create a new Connection (File > New Connection...)
- Connect to host 127.0.0.1, port 1972, click Next:
- Select the detected JVM (you will see activity in the oc command window at this point):
- You can select different views from the "Status" widget and monitor the running JVM:
Instrument the JVM for "Headless mode"
If firewalls and/or security concerns prevent you from directly connecting to a system in socket mode, you can use Health Center in headless mode.
There are two options for enabling this:
- Specify JVM options (requires restarting the pod)
- Attach to an already running JVM
Specify jvm.options for headless mode:
You can instrument the JVM with jvm.options (see jvm.options section
above) and restart the pod.
These example settings will let the agent collect data for 2 minutes then pauses for 5 minutes. This continues for 5 cycles. (see
Health Center configuration properties).
-Xhealthcenter:level=headless-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=dump-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=2-Dcom.ibm.java.diagnostics.healthcenter.headless.run.number.of.runs=5-Dcom.ibm.java.diagnostics.healthcenter.headless.run.pause.duration=5You could also keep running Health Center indefinitely (i.e. until the JVM stops) and control the amount of storage used by specifying the
.files.max.size and
.files.to.keep properties (in this example 5 files with approx. 20MB each):
-Xhealthcenter:level=headless-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=dump
-Dcom.ibm.java.diagnostics.healthcenter.headless.files.max.size=20000000-Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=5You can later collect the *.hcd files from the dump volume (check Persistent Volume
baw-dumpstore-pv in order to find the directory):
Attach to running JVM for headless mode
You can attach to an already running JVM to activate headless mode, e.g. after you have observed a problem at runtime which might disappear after a pod restart. (NOTE: this would not to work for socket mode, because we cannot change the rmi.server.hostname property for the running JVM)
-
-
Open a shell into the pod:
oc rsh icp4adeploy-instance1-baw-server-0
-
Attach Health Center:
JVM_ID=$(pgrep java)
java -jar ${JAVA_HOME}/lib/ext/healthcenter.jar \
ID:${JVM_ID} level=headless \
-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=dump \
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=2 \
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.number.of.runs=5 \
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.pause.duration=5