AIOps: Monitoring and Observability - Group home

Since its inception in 2016, IBM® Z® OMEGAMON® for JVM has leveraged the Java^TM tools component Health Center agent as a source of monitoring data. Health Center is an excellent diagnostic and performance analysis tool in its own right. When used with the Eclipse framework plugin it provides very nice graphical views of JVM performance data for a single JVM at a time. Health Center also has an application programming interface (an API in the traditional sense of the acronym) which customers and vendors can use to consume the same metrics used by the Health Center plugin. The Java Tools team went further by providing a JVM local interface to the monitoring API using the level=inprocess option. OMEGAMON for JVM uses this feature which allows it to monitor many JVMs concurrently without using any network connections.

There were some drawbacks in using the Health Center Agent and monitoring API however. The (pure Java) API was originally designed for use in a remote UI-type environment. It loads many additional classes, including UI helper classes that are not necessary in an event-driven in-process implementation.  It also spawns a lot of threads, useful for multiplexing collection to a socket connected UI, but not of much value for in-process collection where many of these threads are simply not used. In z/OS every thread is also backed by a Language Environment (POSIX) thread and stack frames which results in wasted memory. Finally, Health Center used semaphores for event notification and timing which could not be discriminated in multiple JVM enclaves such as the ones found in CICS TS.

IBM Z OMEGAMON for JVM, V5.5.0 PTF UJ06029 introduces a new monitoring option that does not require Health Center to be configured on the JVM. The new option uses a JVMTI agent and other Java services to collect the same data that Health Center does. 

Elimination of Health Center:

• Reduces the number of additional threads used for monitoring
• Reduces the number of classes required for monitoring
• Eliminates residual temporary files used by Health Center
• Eliminates orphaned semaphores

In turn, these changes reduce the amount of CPU used for monitoring and reduce the virtual memory footprint.

The new OMEGAMON agent option also simplifies configuration. To use the new collection mode, you simply remove the Health Center options from the JVM configuration and instead use the OMEGAMON JVMTI agent with the -agentpath JVM option.

Old JVM options:

-Xbootclasspath/p:/rtehome/KJJ1/kan/hca_64/lib/ext/healthcenter.jar

-agentpath:/rtehome/KJJ1/kan/hca_64/bin/libhealthcenter.so=path=:/rte

home/KJJ1/kan/hca_64,level=inprocess,disableCH

-javaagent:/rtehome/KJJ1/kan/bin/IBM/kjj.jar

New JVM options:

-agentpath:/rtehome/KJJ1/kan/bin/IBM/libkjjagent_64.so

-javaagent:/rtehome/KJJ1/kan/bin/IBM/kjj.jar

For z/OS Connect EE JVMs only, you will still need to include

-Xbootclasspath/p:/rtehome/KJJ1/kan/bin/IBM/kjjboot.jar

Results
The performance improvement is tangible. With 24 fewer threads, and around 500 fewer classes the Java heap memory use is reduced by about 40MB per JVM.  z/OS native memory usage is reduced by about 100MB.  Fixed CPU utilization is also reduced. The reduction varies depending on workload and processor speeds. Let us know what your experience is.
 
Availability
This new feature is available with PTF UJ06029 (APAR OA61482) from https://www.ibm.com/software/shopzseries/ShopzSeries_public.wss

Links

IBM Z OMEGAMON for JVM Product page
https://www.ibm.com/products/z-omegamon-for-jvm
Apply for a free zTrial
https://www.ibm.com/account/reg/signup?formid=urx-19503


Tags