AIOps: Monitoring and Observability - Group home

New! JVM Diagnostic Actions in IBM Z OMEGAMON for JVM V5.5.0 - July 2022 PTF

  
Overview
One of the greatest strengths of IBM Z OMEGAMON for JVM is its ability to capture and summarize performance problems in Java applications before they become a critical issue and an application availability problem. Once a problem is identified,  Java subject matter experts (SME) need to home in on the underlying cause of the issue.  For this, they will often need the diagnostic information that can only be gleaned from dumps: Java core dumps; Java heap dumps or system (SVC) dumps. With PTF UJ08933 (APAR OA63273) released in July 2022, users of OMEGAMON for JVM V5.5.0 can generate any one of these dump types through the enhanced 3270UI.

How to Request a Dump
This new feature adds a new item to the Health Summary workspace action menu in the enhanced 3270UI (e3270UI):  option "U" for "Utility".  Select this on the JVM process row that you wish to diagnose.  It will invoke a popup menu in which you can specify the diagnostic you require:


Select one of the dump options ("p", "c",  or "v") to immediately request the corresponding  type of dump.  If you are authorized to make this request, you will be presented with a popup message confirming that the request has been submitted.  To find the name and path of the requested dump, you will need to review the log of the problem JVM. The JVM will print a message containing the name and path of the requested dump. 


The utility functions are implemented as an "action agent" in the IBM Tivoli Monitoring (ITM) framework.  This is also sometimes known as a "prefix command" agent because a special prefix is used, usually the OMEGAMON product code, when the command is used in an ITM Take Action request or as a reflex action in a situation (see below).  Prefix commands issued from the e3270UI may be protected by z/OS system security (e.g RACF), so that the commands can be issued only by authorized personnel. The RACF class used to protect prefix commands is specified during RTE configuration in PARMGEN or IBM Z Monitoring Configuration Manager (IZMCM). This infrastructure is common to all of the OMEGAMON products and is described in the OMEGAMON Shared Documentation. 

A profile is created for the SAF class defined as the RTE security class with the pattern <Kpp>.<msn>.TAKEACTION where pp is the product code and msn is the managed system name that the command will execute on.  It is usually more convenient to define a generic profile using "**" as the managed system name.  Only the command interface is protected in OMEGAMON for JVM. Authorizing someone READ access to the Kpp.msnTAKEACTION profile will allow them to request any of the dump or TRACE commands.

Using the Utility Functions as Situation Actions
Because the utility functions are implemented as a command prefix agent, they can be used as a reflex action in an ITM situation. Take Java heap occupancy as an example. This metric reports the percentage of heap that is still in use after garbage collection. If this value is more than 80% (after global garbage collections), it could mean that the maximum heap size is too small or that there is a memory leak. A subject matter expert would usually turn to a heap dump to determine whether there is a memory leak. OMEGAMON for JVM includes a product-provided situation for high heap occupancy (JVM_Occupancy_after_GC_Critical). You can add an Action to this situation to automatically take a heap dump when the situation is triggered.

1.  Using the Tivoli Enterprise Portal (TEP) Situation Editor, select the Action tab in the situation definition for JVM_Occupancy_after_GC_Critical.   Select "System Command" for the Action Selection. Begin the command with the command prefix "JJ:" 


2. Click  the Attribute Substitution button to select the attributes that must be passed to the action agent. These are:
Job Name
ASID
ProcessID
Note: The order is important.  Select each of the attributes in turn, pressing the OK button. 


The TEP fills in a variable which will contain the jobname of the JVM when the situation is triggered.


3. After specifying the three attributes required to identify the JVM in question, enter the name of the utility command you want to execute. In this case, use HEAPDUMP.  The situation definition will now look like this:


4. Click the Apply button.
5. Fill in the rest of the situation definition like distribution and situation interval. 
6. Save the situation by returning to the Formula tab and pressing Enter.  
7. Start the situation. 

When the situation triggers, you will see a message in the TEMA agent log that the command was issued:
and a message in the target Java application log:

22/07/26 15:18:11.003 84280050 KJJEX0008I Generating heap dump
JVMDUMP034I User requested Heap dump using '/proj/dumps/heapdump.20220726.161811.84280050.0001.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /proj/dumps/heapdump.20220726.161811.84280050.0001.phd

Process this dump with an offline tool such as the Java heap analyzer.

Links
IBM Z OMEGAMON for JVM Product Overview
IBM Z OMEGAMON for JVM V5.5.0 Documentation


#OMEGAMON
#z/OS
#z/OSConnect