In a previous post on
Java Performance Tools: Sampling Profiler, we covered how to use the Health Center sampling profiler available in IBM Java to investigate Java CPU usage in a low overhead way. One additional question people often have is how to figure out what's driving Java memory allocations. Heapdumps may be analyzed in the
Eclipse Memory Analyzer Tool, but gathering heapdumps will pause the JVM for up to dozens of seconds, they're just a single snapshot in time, and it's also not easy to figure out what callstacks drove the creation of objects.
A lesser known feature of Health Center allows finding out what's driving Java memory allocations over time in a low overhead way.
Health Center Object Allocation Sampling
When you enable Health Center, by default, it will gather
object allocation sampling. The way this works is that every time that approximately X bytes are allocated outside of thread local heaps, the JVM records the class name and the size of the requested object allocation. As of this writing, the value of X
defaults to 16MB. In other words, at approximately every 16MB of Java heap allocations outside of thread local heaps, the JVM samples the object allocation at that point. This value may be controlled with the undocumented option:
-Xgc:allocationSamplingGranularity=X
The lower this value, the higher the accuracy of the sampling but also the higher the overhead.
To view this data in Health Center, click on Garbage Collection and then switch to the "Samples by Object" view:
Finding what's driving memory allocations
While the default view is useful, what we generally want to know is the callstacks driving the memory allocations. In the live socket mode, Health Center allows you to
enable capturing callstacks on object allocation samples through a menu item. However, Health Center is often used in headless mode rather than the live socket mode. When Health Center is enabled in headless mode, it sets various
-Xtrace options to capture most of the data that you see in the client; however, there's no headless option to configure capturing callstacks on object allocation samples in headless mode. All we really need to do is specify the following additional
-Xtrace option which is effectively what the menu option does in the live socket mode:
-Xtrace:trigger=tpnid{j9mm.395,jstacktrace},stackdepth=5
However, we can't specify -Xtrace options along with -Xhealthcenter:level=headless because headless mode overrides all specified -Xtrace options.
Instead, what we can do is dynamically enable the additional -Xtrace options at runtime after Health Center has started. What we need to do is get code inside the JVM that calls the static Trace engine set API method after the JVM has started with Health Center in headless mode:
com.ibm.jvm.Trace.set("trigger=tpnid{j9mm.395,jstacktrace},stackdepth=5")
One way to do this is to use Java Surgery. This uses late attach to inject code into a running JVM and the latest version of Java Surgery includes a built-in command to call com/ibm/jvm/Trace.set. After the JVM has started with Health Center in headless mode, execute Java Surgery with the following arguments, replacing $PID with the process ID of the JVM:
java -jar surgery.jar -pid $PID -command J9TraceSet "trigger=tpnid{j9mm.395,jstacktrace},stackdepth=5"
Note: There is a
known issue on some JVMs that causes problems with
jstacktrace. In those cases, also add
,none=j9vm.682-683.
Gather and load the Health Center HCD files as you normally would and now, when you click on a class row in the "Samples by object" view, the "Call hierarchy" view at the bottom shows aggregated call stacks driving the selected object allocation samples:
If you zoom into a particular time period in other views of Health Center (such as the CPU view), then these object allocation samples will be cropped to the same period.
Summary
In conclusion, Health Center may be used to find what's driving Java memory allocations by callstacks over time in a very low overhead way. You may do this through a menu option in the live socket mode, or, if you're using the Health Center headless mode, you may use Java Surgery or a similar mechanism to enable the required
-Xtrace options at runtime after the JVM has started.
#app-platform-swat#automation-portfolio-specialists-app-platform#Java#Liberty#performance#websphere#WebSphereApplicationServer(WAS)