WebSphere Application Server & Liberty

If you experience production issues with WebSphere Application Server (WAS) traditional, it's often useful to log performance statistics to a file for later analysis. This includes things such as thread pool utilization, application response times, and so on. Running without such monitoring data is like flying an airplane with some of the gauges broken.

Some of the major ways of monitoring WAS performance statistics are:

• IBM and 3^rd party monitoring products such as Instana
• Prometheus and Grafana integration (or an HTTP client) using metrics.ear since WAS 8.5.5.20 and 9.0.5.7
• Live monitoring with the administrative console's Tivoli Performance Viewer (TPV)
• The Performance Tuning Toolkit (PTT) (unsupported tool)

All of the above techniques ultimately get their data from the WAS Performance Monitoring Infrastructure (PMI). This is the component inside WAS that gathers configured statistics aggregated over approximately 30-second intervals, and makes this data available to the monitoring tools. By default, PMI is configured in "Basic" mode which gathers core statistics and has an overhead of about 2%.

Logging PMI data to files

If you don't use an IBM or 3^rd party monitoring product, or Prometheus, the rest of this article shows you how to log performance data to files as part of the built-in WAS product. Even if you do use 3^rd party monitoring products that already log such data, this technique may still be useful because IBM Support generally does not analyze the exported proprietary data formats from 3^rd party monitoring products.

Prerequisites

First, ensure PMI is enabled. PMI is enabled by default.

1. Navigate to each application server's PMI configuration:
   
   Administrative Console } Servers } Server Types } WebSphere application servers } $SERVER } Performance } Performance Monitoring Infrastructure (PMI)
   
   
2. Ensure "Enable Performance Monitoring Infrastructure" is checked

On the same page, configure PMI configuration for your application needs. The default is "Basic" mode with generally useful statistics. You may consider using "Custom" mode with additional statistics based on your application usage.

Starting and stopping PMI logs through the administrative console

The following instructions start and stop PMI logs that are written to XML files in $WAS/profiles/$PROFILE/logs/tpv/

1. Administrative Console } Monitoring and Tuning } Performance Viewer } Current Activity
2. Select all application servers you want to log and click "Start Monitoring"
3. Click each monitored application server link and:
   1. Click on $SERVER } Settings } Log
      1. Duration = 999999
      2. Maximum File Size = 50
      3. Maximum Number of Historical Files = 5
      4. Log Output Format = XML
      5. Click Apply
   2. Click $SERVER } Summary Reports } Servlets
   3. Click "Start Logging"
      1. Note that this starts logging for all PMI data, not just for servlets
4. Reproduce the problem
5. To stop logging, go back into the links above and click "Stop Logging" for each server

Note that there is no way to automatically start PMI logging when a JVM is started. If desired, logging must be restarted after each JVM restart.

Viewing PMI logs

The administrative console includes a function that reads and replays the PMI XML files gathered above:

1. Administrative Console } Monitoring and Tuning } Performance Viewer } View logs
2. Choose the XML file from your computer or from the server
3. Click "View Log"
4. Expand "Performance Modules" and check the metrics you would like to see
5. Click "View Module(s)"
6. In the following example, we can see a spike in the WebContainer average concurrently active threads (ActiveCount) at about 4:22PM:

This viewer is the same as what you would see when live monitoring with the addition of play, pause, fast-forward, and rewind buttons at the top (the above screenshot only shows the rewind button as the end of the file has been reached). By default, the viewer starts in play mode and iteratively shows more data in simulated real time.

Note that you do not need to use the same administrative console to view the data as the cell that produced it, although it's best to use the same version and fixpack of WAS if possible. This technique allows you to gather PMI data in production and load it in a pre-production cell to avoid adding overhead to production.

Discussion

WAS PMI provides aggregated statistics for configured metrics that help you understand the overall behavior and performance of your applications. WAS TPV PMI logging may be used to log such statistics to files which may replayed through an administrative console or shared with IBM Support (when investigating potential product defects).

Note that this discussion focuses on low-overhead, aggregated performance statistics. This approach is particularly useful to investigate general health issues and during performance tuning exercises. For more narrow diagnostic situations, alternatives are usually easier to use such as thread dumps, hung thread detection, diagnostic plans, request metrics, sampling profilers, database timing, HTTP access logs, and so on.

Starting and stopping PMI logs through wsadmin

The below example wsadmin script is provided as-is and without any support and it may be used to automate starting and stopping PMI logging.

---

See our team's previous post in the Lessons from the field series: IBM Java and OpenJ9 Just-In-Time Compiler Tuning


#app-platform-swat
#automation-portfolio-specialists-app-platform
#websphere
#WebSphereApplicationServer(WAS)