This article shows how the IBM Health Center can be easily installed into the CICS Explorer and used to create customized perspectives for monitoring CICS JVM servers on z/OS. Together these two tools provide a powerful platform for profiling Java applications, monitoring JVM memory usage and garbage collection, and performing further diagnostic commands.
Requirements
To follow the setup in this article you will require the following minimum software requirements:
- CICS TS for z/OS V4.2 or later
- IBM Java SDK for z/OS V7.0 for SR6 (or later)
- A Windows or Linux client with the CICS Explorer or z/OS Explorer installed, using a supported IBM Java SDK.
The configuration in use in this article was as follows:
- CICS TS V5.2 open beta
- IBM Java SDK for z/OS V7.1 SR0
- IBM Monitoring and Diagnostic Tools for Java - Health Center -v2.2.0.201401101227
- Windows 7, with CICS Explorer V5.2.0.0 open beta and IBM Java SDK 7.0 for Windows
What is the IBM Health Center?
The IBM Health Center is a diagnostic tool for monitoring the status of a Java Virtual Machine (JVM) and can be used with any IBM JVM including CICS JVM servers. It has a low runtime overhead and provides a highly customizable client as part of the IBM Support Assistant (ISA) workbench. Since the Health Center client is Eclipse based, it can also be installed into other Eclipse environments such as the IBM CICS Explorer.
Installing the Health Center
The first step is to install the Health Center plugin into your CICS Explorer Rich Client Platform (RCP). To do this you will need to add a new software repository to your CICS Explorer, as follows:
- In the CICS Explorer, click Help -> Install New Software
- Click Add to add a new software repository and provide the following location information:
- Next click the drop-down box labeled Work with, and select the site that was just added in step 2. This will validate that the Health Center is available in the repository specified.
- Click Next in the ` window to confirm the Health Center Core Feature will be installed.
- Review the license information and click Next.
- Click OK to confirm the installation of unsigned content, and then click Yes to restart the CICS Explorer when prompted.
Note if you have installed CICS Explorer using IBM Installation Manager, the Health Center download is now integrated into the Aqua download side, just click Install -> and then search for “Health Centerr” and you will see the Health Center available under the IBM Monitoring and Diagnostic Tools.
Configuring the JVM for monitoring
To enable a JVM for monitoring it is necessary to add a few JVM arguments to enable the Health Center agent and the TCP/IP port it will listen on. When using a CICS JVM server the JVM arguments are set in the JVM profile which is a configuration file stored in zFS.
-
To edit this file in CICS Explorer, click Window – > Open Perspective -> z/OS.
This will open a new Eclipse Perspective, which is a customizable set of views and includes the CICS Explorer z/OS UNIX Files view for editing zFS files. However, before you can access z/OS you will first need to define an FTP connection to your z/OS system. To do this in the Host Connections view expand z/OS -> z/OS FTP and click Add.
- In the Host name field, enter the hostname of the FTP server for the z/OS system and click Save and Connect. This will then display the Signon window. Enter a z/OS user ID and password as credentials for the FTP connection and click OK to connect to the FTP server
- Now using the CICS Explorer z/OS UNIX Files view navigate to the location of the zFS directory used to store the JVM profiles for this CICS region. In our example we are using DFHOSGI.jvmprofile stored in the
/var/cicsts/dfhconfig/JVMProfiles/ directory. The name of the JVM profile used by a JVM server is configured in the JVMSERVER resource definition using the JVMProfile attribute which sets the prefix, and the suffix is always .jvmprofile. The directory is set using the JVMPROFILEDIR SIT parameter.
-
Add the following -Xhealthcenter JVM argument, this enables the Health Center agent to collect monitoring data and specifies the listening port. We specified the port number 8125
-Xhealthcenter:port=8125
Health Center also uses an additional listening port for its IIOP connection. If you want to control which ports are used ahead of time then you can add the following system property which sets the additional IIOP port used by the Health Center.
-Dcom.ibm.java.diagnostics.healthcenter.agent.iiop.port=8126
Alternatively you can specify the use of the JRMP protocol (instead of IIOP) for connecting the Eclipse client to the Health Center agent. This can be useful if you experience problems connecting your Java client to the Health Center agent on the server.
-Dcom.ibm.java.diagnostics.healthcenter.agent.transport=jrmp
If you are on a z/OS system with multiple IP hostnames and experience problems connecting to the health centre agent you can use the following property to set the hostname used by the Health Center.
-Djava.rmi.server.hostname=<hostname>
Your CICS JVM server profile should now look similar to the following:
-
To enable the new JVM settings it is necessary to stop and restart the JVM. This can be done using the CICS Explorer JVM Servers view as follows:
- In CICS Explorer click Window –> Show View -> Others, select JVM Servers and click OK.
- Select the CICS region in the CICSplex Explorer view, this will show the state of the JVM servers in this CICS region.
-
Select the JVM server and click Disable -> Phase Out and then click OK
- Select the JVM server again and click Enable and then OK to start the JVM.
Once the JVM server is started the Health Center agent will initialize and immediately start buffering monitoring data ready for collection by the Health Center client. At this stage you can check the Health Center agent is listening on the TCP/IP port using the following MVS TSO command
NETSTAT (PORT 8125
In addition if the JVM server stderr file is viewed, this will show log messages similar to the following indicating the Health Center agent was successfully started and is listening on the configured ports.
Dec 31, 2013 1:16:48 PM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean <init>
INFO: Agent version "2.2.0.20131003"
Dec 31, 2013 1:16:48 PM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startMBeanServer
INFO: IIOP will be listening on port 8126
Dec 31, 2013 1:16:49 PM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent
INFO: Health Center agent started on port 8125.
Connecting the Health Center client to the JVM
Now that the JVM server is running you can use the Health Center views to monitor the CICS JVM server. The Health Center Environment perspective provides a convenient way to access all the Health Center function. It can be simply accessed using Windows -> Open Perspective -> Other -> Health Center Environment
- To connect to the Health Center agent running in the JVM you have to run the connection wizard. This can be accessed from any of the Health Center perspectives using File -> New Connection menu. This starts the Health Center connection wizard.
- Click Next and then enter details for the listening port, 8125, as specified previously in the CICS JVM profile.
- Click Next and this displays the following window to confirm the port is being used by a Health Center agent on the host.
The Status view in this perspective can now be used to access the different functions of the Health Center, such as CPU, Environment, Garbage Collection, Locking, Native Memory, Profiling and Threads. The next section will discuss how some of these can be used to analyze the CICS JVM server environment.
Garbage collection
The Garbage Collection perspective provides a set of views to assist in analyzing the garbage collection (GC) process used by the JVM to manage memory in the JVM heap. Using the default gencon GC policy splits the Java heap into two areas, the new or nursery area, and the old or tenured area. CICS JVM server statistics call new or nursery activity minor GC and call old or tenured activity major GC. The Summary view provides detailed information on the GC process, however, much of this information is also displayed in the CICS Explorer JVM Servers view, which can be added to the Garbage Collection perspective using Window -> Show view -> Other –> JVM Servers. In addition individual Health Center graphs for Used heap and Pause time can also be added using the same Show view menu, and a customized perspective created with a combination of both Health Center and CICS views as shown below.
Custom perspectives such as these can be saved for future use by right-clicking on the current perspective and selecting Save As, which will add the perspective to the list of available perspectives on the top row.
In our example above, you can see that the initial heap of 30MB provides adequate space for the current memory usage, and the GC process is able to minimize the heap occupancy to approximately 6.8MB each time it is invoked. Out of the 949 garbage collections events that have run, all but 5 were in the nursery area (minor), which indicates most garbage is short lived objects from the CICS transaction invocations.
One of the most useful GC views provided by the Health Center is the Analysis and Recommendations view. This provides analysis for each perspective, and is a simple way of analyzing heap usage. In the Garbage Collection perspective above, the following analysis was provided, which relates accurately to the increase in workload used to generate this test.
Profiling
The Profiling perspective allows method level profiling of the applications running within the JVM. The methods can be filtered by class or package name. In the figure below we can see that during 23.1% of the samples taken, the method that was running was called from the main method of the JavaLinker class which is defined as the entry point for our CICS program. In addition the formatTime() method was active in 3 samples out of 44, and so could be considered as a candidate for optimization.
Additionally the Invocation and Called method views allows you to analyze the call path of each profiled method to ascertain how it was invoked, and what further methods it calls.
Threads
The Threads perspective provides detailed analysis of all the threads running within the JVM. The Current threads view supplies a useful Thread name filter box, which allows the display to be filtered. Using a filter of *.TASK.* will locate the CICS task threads which are dubbed with the pattern program.task.tranid. For each thread displayed it is possible to show the call stack, this will show where each thread is currently suspended. In our example we can see that CICS task 53525 is running under transaction ID CSMI and is currently suspended in a Thread.sleep() method called from the main method of the JavaLinker() class.
Customizing data collection
Instead of requiring a socket connection to a monitored JVM, Health Center can save the data that it is analyzing to a number of .hcd files on the local machine. These files can then be opened in the Health Center Eclipse client at a later date, without the need for a live connection. This is termed headless mode, and is useful for systems where you cannot connect a client to the agent. For example, where firewall restrictions prevent connection.
To enable headless mode set the following property in the CICS JVM profile.
-Dcom.ibm.java.diagnostics.healthcenter.data.collection.level=headless
To control the directory location of the temporary files used to buffer data for headless collection you can set the following system property. In the example below this is set to a zFS directory unique to the specific CICS JVM server using the &APPLID and &JMVSERVER substitution symbols which will be substituted at runtime by the relevant CICS values.
-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/cicsjava/logs/&APPLID;/&JVMSERVER;
Additionally you should also set the run duration (in minutes) and a number of runs, this will enable you to collect output files after a set period of time.
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=10
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.number.of.runs=2
In this directory you will see a set of files created as shown below, and then after the duration has expired a .hcd suffixed file will be created.
$> ls /cicsjava/logs/IYK2Z32E/OSGIJVM1
ClassHistogramSource67174597_1
MemoryCountersSource67174597_1
MemoryCountersSource83952787_2
MemorySource67174597_1
MemorySource83952787_2
MethodDictionarySource67174597_1
MethodDictionarySource83952787_2
TRACESubscriberSource67174597_1
TRACESubscriberSource83952787_2
ThreadsSource67174597_1
ThreadsSource83952787_2
healthcenter20140103_023244_83952787_1.hcd
To analyze the run transfer the .hcd suffixed file to your workstation in binary mode and open using the File -> Load Data menu in the Health Center client, which will then enable analysis of the run using any of the facilities described.
If the data that Health Center collects exceeds the maximum file size of 2GB the agent automatically creates a second file, and so on. If you are gathering lots of data, a long headless run will result in the collection of multiple files. By default, Health Center keeps only the last 10 files. You can override this value by using the following system property.
-Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=<n>
Alternatively to minimize data collection in a JVM that is not being monitored you can start the agent but delay collection until a client connects. This means there is no impact on the application that is running, until the client actually connects to the agent. To enable this mode set the following JVM property.
-Dcom.ibm.java.diagnostics.healthcenter.data.collection.level=off
Summary
This article has shown how the functionality of CICS Explorer for monitoring Java workloads in CICS TS, can be easily extended using the IBM Health Center plugin. Let us know how you get on and what further information you are interested in.
References
If you wish to explore this area further the following articles and papers are useful references: