WebSphere Application Server & Liberty

For a video presentation of this topic, see the replay of the Customer Advisory Board session.

Gauges: Small Airplanes, Large Airplanes and Space Shuttles

Fifteen years ago, most administrators had very limited monitoring of their WebSphere Application Server (WAS) environments. Many only watched CPU and memory on the node, and Java heap usage of WAS. Some customers still run like this today.

These gauges are insufficient. It’s akin to flying an airplane with half the gauges broken. A pilot can still look out the window and it might usually work, but at nighttime or in bad weather, lack of proper gauges can be dangerous when flying due to issues like spatial disorientation. Running critical business infrastructure requires more sophisticated monitoring akin to a large, modern airplane with all the right gauges and alerts.

Today, most customers have moved in the direction of more sophisticated monitoring, but this has led to different problems:

1. Enabling too many gauges makes WAS more like a space shuttle than an airplane, and this is usually overkill that causes confusion.
2. Despite the availability of the necessary gauges, critical gauges are missing from dashboards, or they’re not watched or alerted on.
3. Administrators sometimes become overwhelmed during incidents due to poor dashboards, not knowing which gauges to look at, and poor tools, processes, and automation to properly respond.

This article discusses the critical gauges to use to monitor WebSphere environments, how to configure alerts for gauge thresholds, and outlines processes and automation to help keep environments running smoothly.

Deployment Size

First, decide which type of deployment you’re targeting: a small airplane, a large airplane, or a space shuttle. In the vast majority of cases, we recommend targeting a large airplane, even if you might think your application or deployment is intuitively “small”. The reason is that there is little marginal cost to enabling the gauges of a large airplane and they are often valuable.

The following are minimum gauges for small and large airplane deployment targets for the most common types of HTTP-based applications. Additional metrics apply for JMS, EJB, etc. The term “node” is used to refer to the operating environment of a WAS process such as a physical server, virtual machine, or cloud container.

Small Airplane Deployment

Minimum end-to-end gauges:

1. Average response time
2. Total throughput

Minimum gauges for each node:

1. CPU utilization
2. Memory utilization (excluding file cache)
3. Operating System Logs’ Warning and Error counts

Minimum gauges for each WAS Java process (JVM):

1. Thread pool utilization
2. Connection pool utilization
3. Java heap utilization
4. WAS Logs’ Warning and Error counts (System*/messages*, native*/console*, ffdc)

Minimum gauges for each reverse proxy web server:

1. Connection counts
2. Logs’ Warning and Error counts (error_log*, http_plugin.log)

Large Airplane Deployment

The large airplane deployment is the generally recommended minimum set of gauges.

Minimum end-to-end gauges:

1. Average response time
2. Maximum response time
3. Total throughput
4. Error throughput (e.g. HTTP code >= 400)
5. Application-specific metrics
6. Request arrival rate

Minimum gauges for each node:

1. Is it alive and responsive?
2. CPU utilization
3. Memory utilization (excluding file cache)
4. Network utilization
5. Disk utilization and average service time
6. Filesystem consumption
7. Count of TCP retransmissions on LAN interfaces
8. Operating System Logs’ Warning and Error counts

Minimum gauges for each WAS Java process (JVM):

1. Is it alive and responsive?
2. JVM process CPU utilization
3. Thread pool utilization
4. Connection pool utilization
5. Java heap utilization
6. Proportion of time in garbage collection
7. Average database response time
8. Average HTTP response time
9. WAS Logs’ Warning and Error counts (System*/messages*, native*/console*, ffdc)

Minimum gauges for each reverse proxy web server:

1. Connection counts including breakdown by connection state
2. Thread pool utilization
3. Access log with response time
4. Logs’ Warning and Error counts (error_log*, http_plugin.log)

WAS Gauge Data 

WebSphere Application Server exposes gauge data through these major mechanisms:

WAS traditional:

• Performance Monitoring Infrastructure (PMI)
• Prometheus style /metrics HTTP endpoint through APAR PH24409 (since 8.5.5.20 / 9.0.5.7)

Liberty:

• Prometheus style /metrics HTTP endpoint through mpMetrics
• Java MXBeans through monitor
• JAX-RS OpenTracing with Jaeger, Zipkin, etc. through mpOpenTracing

Most commonly, this gauge data is integrated into monitoring products. Speak with your IBM account representative for details.

Alerts

Slow Application Processing Detection

WebSphere Application Server provides built-in slow application processing detection. When the threshold is breached, a warning is printed to WAS logs and may be detected by the warning and error gauge.

In general, we recommend setting the threshold (X) to the maximum expected response time (in seconds) plus 20%.

Configure WAS traditional hung thread detection and watch for WSVR0605W warnings:

• ibm.websphere.threadmonitor.interval=1
• ibm.websphere.threadmonitor.threshold=X

Configure the Liberty requestTiming feature (if needed, tune sampleRate to reduce overhead) and watch for TRAS0112W warnings:

<featureManager><feature>requestTiming-1.0</feature></featureManager>
<requestTiming slowRequestThreshold="Xs" sampleRate="1" />

Gauge Alerts

Below are some ideas for gauge alerts which should be tuned to your particular environment, service level agreements, and applications. Many points make reference to a “historical average” which implies “for that time of day, day of week, season, etc.” Alternatively, you may choose simple thresholds based on observations, SLAs, etc.

Note that often the alert uses != rather than >. For example, average and maximum response times != historical average. Intuitively, you might think that you only have to check for response times exceeding a threshold; however, if they are excessively low, this may also be a symptom of a problem such as errors in the backend that are quickly returning errors to the user (e.g. HTTP 500).

End-to-end gauge alerts:

1. Average and maximum response times != historical average for > ~1 minute
   1. For systems that are responding to humans, ideally, a nice target maximum is 400ms which is approximately the human perception threshold (Herzog et al., 2016)
2. Request arrival rate != historical average for > ~1 minute
3. Total throughput != historical average for > ~1 minute
4. Error throughput != historical average for > ~1 minute

Per-node gauge alerts:

1. CPU utilization > ~90% for > ~5 minutes
2. RAM usage (excluding file cache) > ~90%
3. Disk utilization > ~90% for > ~1 minute
4. Disk service time > ~10ms
5. Filesystem consumption > ~90%
6. Network utilization > ~90%
7. Network ping time on a LAN > ~1ms
8. Network TCP retransmits on a LAN != 0

Per-JVM gauge alerts:

1. WAS traditional application thread pool usage > ~90% for > ~1 minute
2. Liberty thread pool usage != historical average for > ~1 minute
3. JDBC connection pool > ~90% for > 1 minute
4. JDBC average response time > historical average for > ~1 minute
5. Java garbage collection proportion of time in GC > ~10% for > ~1 minute
6. Java heap utilization > ~90% of -Xmx for > ~1 minute
7. WAS traditional logs’ warning and error counts from ffdc, " W " or " E " messages in System*log, and “JVM” messages in native*log > historical average for > ~1 minute
   1. There should be a separate alert for slow application processing detection warnings (WSVR0605W)
8. Liberty logs’ warning and error counts from ffdc, " W " or " E " messages in messages*log, and “JVM” messages in console.log > historical average for > ~1 minute
   1. There should be a separate alert for slow application processing detection warnings (TRAS0112W)

Per-reverse proxy web server gauge alerts:

1. Connection counts != historical average for > ~1 minute
2. Thread pool usage != historical average for > ~1 minute

Where is the problem?

Modern airplanes have advanced gauges, gauge alerts, self-correction, and auto-pilot capabilities; however, sometimes a human is needed to take manual control or hypothesize where a complicated problem may be, especially in emergencies or when many gauge alerts are going off at once. Similarly, the above gauges and gauge alerts are a good way to find many problems, but a problem determination methodology is still needed, especially for emergencies or very complicated problems.

One of the most common complicated problems is determining the cause of poor performance or a bottleneck. This is particularly true in cloud and microservice architectures which have many tiers and backend services.

For such problems, it’s useful to apply a bit (but not too much!) of queuing theory from computer science. There are three key proximate variables in evaluating performance at each tier:

1. Request arrival rate
2. Processing concurrency
3. Response time

These are indirectly affected by many variables. For example, CPU saturation will impact response times, etc. Nevertheless, these three variables are the foundation of performance.

It’s useful to have a mental model like a toll road with each tier (e.g. WebSphere, Database, etc.) of toll booths (application threads) serving your user requests (cars):

Throughput per tier is how many cars are processed per unit time per tier and that will depend on the number of toll booths (concurrency/threads), arrival rate of cars, and average toll booth processing time (response time). Except for the first tier, the arrival rate of downstream tiers (e.g. database) is a direct result of the throughput of the previous tier (e.g. WebSphere) and this helps reason about where a problem is. Conversely, a slow down at one tier of toll booths (e.g. database) may affect the upstream toll booths (e.g. WebSphere).

The analogy isn’t perfect since a real-life toll road has different starting and destination points. Just imagine that when a request takes its “offramp” (which might be after the final tier, or it might be between tiers due to caching, etc.) it’s looping back “home” to the requesting computer.

Imagine you’re a toll road administrator and you hear that there’s a problem. There’s a big back-up of cars at the database tier of toll booths. What are you first thoughts?

If you were to call or walk up to each toll booth at the database tier in the image above and ask each operator what the holdup is, that’s like a thread dump and it helps you hypothesize what’s slow:

• Is something causing the toll booth operators to perform work slowly? Maybe their electronic cash register is being slow (e.g. high CPU utilization)? In the above example, the WebSphere threads are backed up because the database tier is backed up.
• Check how many live toll booths there are. Did a lot more than average call out of work? Add backup toll booth operators if available.
• Is there a sudden burst of traffic on the road? Maybe there was some sort of event? Can you divert some traffic, add toll booth operators, or make them faster?

Procedures

There are also many non-technical procedures that help you keep your environments running smoothly, including:

1. Having an up-to-date infrastructure diagram to help reason about problems
2. Using a proper performance testing strategy before rolling out application changes
3. Using a maintenance strategy to reduce downtime
4. Performing a failover strategy in emergencies
5. Executing standardized runbooks and automation

Automation is worth dwelling on because I’m constantly amazed at how much time highly paid engineers spend on mundane or repeatable tasks. In addition to labor costs, this dramatically impacts time-to-resolution and serviceability as they often can’t manually get sufficient diagnostics across many servers at the right time. Manual actions simply don’t scale. No matter how good an engineer is, getting thread dumps on even handfuls of servers is often impractical, not to mention that this often involves complex security procedures and jumpboxes and humans can make mistakes.

Automation is a critical component to keep environments running smoothly. It often has a high return on investment, freeing developers and site reliability engineers. Playbooks are a good first start, but it’s important to fully automate or script the actions themselves including gathering diagnostics, transferring logs, etc. This is often done by having a central “automation” node on the same LAN as the target nodes, but this should be planned with your security team as they may have guidelines on things such as password protected SSH keys and other automation techniques.

IBM WebSphere Automation for IBM Cloud Pak for Watson AIOps

With the value of automation in mind, IBM WebSphere Automation for IBM Cloud Pak for Watson AIOps (WebSphere Automation) aims to help with automation of WebSphere Application Server traditional and Liberty estates to quickly unlock value with increased security, resiliency and performance. An example of this value is a single dashboard of WAS installations and unresolved security CVEs, vulnerability tracking, and security bulletin notifications. Speak with your IBM account representative for details.

Summary

In sum:

1. Make sure you’re looking at the right gauges.
2. Set up alerts when gauges exceed thresholds.
3. Use a mental model of the architecture to hypothesize where a complicated problem is.
4. Automate as much as possible.

Appendix

Space Shuttle Deployment

For the few deployments that require more advanced gauges, consider the following additional items:

For each node:

1. System/user/wait/nice/etc. CPU utilization
2. CPU context switches
3. CPU cache misses
4. CPU migrations
5. Interrupt counts
6. CPU and memory pressure statistics
7. Paging rates
8. Per-CPU hyperthread utilization
9. Periodic On-CPU stack sampling
10. Socket counts by state
11. Network counters over time (e.g. delayed ACKs)
12. Network usage by process ID
13. DNS response times

For each WAS Java process (JVM):

1. Per-thread CPU utilization
2. CPU utilization for GC threads
3. CPU utilization for JIT threads
4. Request queue time
5. Global GC pause time
6. Nursery GC pause time
7. Size of requested object driving GC
8. Access log with response time including time-to-first response byte
9. Average HTTP response bytes
10. Per-servlet/JSP average and maximum response times
11. Live HTTP session count
12. Rolled-back transaction count
13. Prepared statement cache discard count
14. Lock contention rates
15. Deadlock count
16. TLS handshakes