WebSphere Application Server & Liberty

 View Only

Lessons from the field #16: Tune Java in-memory caches with heapdumps

By Kevin Grigorenko posted Tue April 19, 2022 08:00 AM

  
Caching is the process of saving the result of a calculation and potentially re-using it in the future to reduce response times, increase throughput, and reduce load on servers. Caching is a critical and often under-tuned aspect of performance. In this article, we'll cover how to understand Java in-memory caches with heapdumps.

Common Caching

When it comes to WebSphere Application Server traditional and Liberty, caching is most commonly done in the following areas:

  1. The application may set HTTP response caching headers. These instruct the user's browser to cache certain content. These headers may also be used by caching proxy servers and/or Content Delivery Networks (CDNs) to deliver cached content closer to each user.
  2. Servlets may be cached in whole or in part using servlet caching in WAS traditional and Liberty.
  3. Java in-memory caches may be used. Common examples include HTTP sessions, authentication caches, Hibernate caches, application caches, and so on.
  4. Enterprise caching products may be used (e.g. with JCache) as side caches or in-line caches for backend servers such as databases and web services.
In this article, we're going to focus on item 3: Java in-memory caches.

Java In-memory Caches

There are too many different types of Java in-memory caches to cover; instead, this article will demonstrate how to use Java heapdumps and the Eclipse Memory Analyzer Tool to explore any type of Java in-memory cache.

A heapdump is a file that's a snapshot of all Java objects in the Java heap. Requesting a heapdump pauses the Java process while the dump is written and this may take dozens of seconds, so consider this performance impact before gathering a heapdump in production. If cache entries are long-lived and the system is not consistently used 24/7, consider taking a heapdump during off-peak hours (e.g. during the night). Heapdumps are generally large files in the range of GB (proportional to the maximum heap size), so ensure you have sufficient disk space before requesting heapdumps. Finally, some types of heapdumps include memory content which may include sensitive user information, so review how to handle such heapdumps with your security team. When possible, gather dumps with memory content as it allows more in-depth analysis.

There are different types of heapdumps and different methods to request them depending your Java vendor and version. For recent versions of IBM Java and Semeru/OpenJ9, there are Portable Heapdumps (PHDs) which do not contain memory content and there are operating system core dumps (Linux/AIX/IBMi=core, Windows=minidump, z/OS=SVCDUMP) which do contain memory content. In general, we recommend using core dumps if possible (also called System Dumps even though they are process dumps). There are various techniques to request IBM Java and Semeru/OpenJ9 heapdumps. For HotSpot Java, there are HPROF dumps which include memory content and there are various techniques to request them.

Once you've collected a heapdump from a representative time when you think a cache of interest should have been filled up, there are various tools available to use to analyze heapdumps. We will be focusing on the free Eclipse Memory Analyzer Tool.

Eclipse Memory Analyzer Tool

The Eclipse Memory Analyzer Tool (MAT) by default only reads HotSpot HPROF dumps. If you are reading IBM Java or Semeru/OpenJ9 dumps, then use the IBM build of MAT that includes DTFJ plugins to read such dumps. There are two downloads: a version of MAT for reading dumps produced by IBM Java <= 8, and a version of MAT for reading dumps produced by IBM Java > 8 or Semeru/OpenJ9.

Note that MAT requires Java 11 on your PATH to run.

In the following example, we will be using a core dump produced by IBM Java 8 which can be downloaded if you'd like to follow along (you'll need to first extract the file with 7-zip).

Finding the cache

The first step in understanding a Java in-memory cache is finding it. If you don't know the class of the cache, then you can start by simply exploring the dump to see if it stands out. If it doesn't, then you'll need to consult the development team for the cache to understand what to look for.

When exploring a dump to see if a cache stands out, the pie chart on the main MAT screen will often show it. This pie chart shows the largest objects from the dominator tree. The dominator tree is a transformation of the Java object graph that partitions mutually exclusive chunks of memory based on object retained set relationships.

In this example dump, the dominator tree pie chart shows one huge chunk of memory which was the cause of an OutOfMemoryError, and a second chunk of memory of an Apache Derby in-memory database cache. Hover over each pie slice to see the object class and address:


Left click on a pie slice and choose List objects } with outgoing references for details on the object. In this example, we see an instance of org.apache.derby.impl.services.cache.ConcurrentCache at the address 0xf28eae20 which is retaining about 8MB of Java heap:

Class Name                                                                              | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------
org.apache.derby.impl.services.cache.ConcurrentCache @ 0xf28eae20                       |           56 |     7,992,320
|- <class> class org.apache.derby.impl.services.cache.ConcurrentCache @ 0xf1b64660      |           80 |           184
|- name java.lang.String @ 0xf1b64630  PageCache                                        |           16 |            48
|- holderFactory org.apache.derby.impl.store.raw.data.BaseDataFileFactoryJ4 @ 0xf28eae90|          184 |         1,912
|- replacementPolicy org.apache.derby.impl.services.cache.ClockPolicy @ 0xf28eaf48      |           32 |         4,600
|- mbean javax.management.ObjectName @ 0xf28ec2b8                                       |           40 |           504
|- cache java.util.concurrent.ConcurrentHashMap @ 0xf28eee78                            |           64 |        32,280
|- hits java.util.concurrent.atomic.AtomicLong @ 0xf28eeeb8  0                          |           16 |            16
|- misses java.util.concurrent.atomic.AtomicLong @ 0xf28eeec8  0                        |           16 |            16
|- evictions java.util.concurrent.atomic.AtomicLong @ 0xf28eeed8  0                     |           16 |            16
'- cleaner org.apache.derby.impl.services.cache.BackgroundCleaner @ 0xf28eeee8          |           32 |           624
-----------------------------------------------------------------------------------------------------------------------

This already gives us some feeling about the cache: 8MB is pretty small so perhaps there's a way to increase the cache size?

One place to start is to understand the number of entries in the cache. Although we can try directly explore the object graph by expanding the fields of the cache, this might get complicated quickly. Instead, once approach is to right click on the cache object and select Show Retained Set and click Finish. This will show a class histogram of what makes up the 8MB of retained heap. Here are the top 5 rows in the default view:

Class Name                                                          | Objects | Shallow Heap
---------------------------------------------------------------------------------------------
byte[]                                                              |   3,738 |    4,036,768
org.apache.derby.impl.store.raw.data.StoredRecordHeader             |  50,630 |    1,620,160
org.apache.derby.impl.store.raw.data.RecordId                       |  40,509 |      972,216
org.apache.derby.impl.store.raw.data.StoredRecordHeader$OverflowInfo|  13,440 |      322,560
org.apache.derby.impl.store.raw.data.StoredRecordHeader[]           |     956 |      284,000
---------------------------------------------------------------------------------------------

This is interesting but often won't get us closer to understanding the cache. However, we can often get a better picture by understanding the retained heap relationships. Click on the calculator icon at the top and select Calculate minimum retained size (quick approx.). Once that completes, sort by the new Retained Heap column in descending order. Here are the top 5 rows:

Class Name                                               | Objects | Shallow Heap | Retained Heap
--------------------------------------------------------------------------------------------------
org.apache.derby.impl.services.cache.ConcurrentCache     |       1 |           56 |  >= 7,992,320
org.apache.derby.impl.services.cache.CacheEntry          |   1,000 |       32,000 |  >= 7,900,440
org.apache.derby.impl.store.raw.data.StoredPage          |     946 |      204,336 |  >= 7,767,736
byte[]                                                   |   3,738 |    4,036,768 |  >= 4,036,768
org.apache.derby.impl.store.raw.data.StoredRecordHeader[]|     956 |      284,000 |  >= 3,198,936
--------------------------------------------------------------------------------------------------​

This is becoming more interesting. One thing that jumps out is that the retained heap is largely consumed by 1,000 instances of org.apache.derby.impl.services.cache.CacheEntry. To understand what's directly referenced by the CacheEntry objects, right click on that row and select Show objects by class } by outgoing references and explore.

Despite us not knowing anything about the details of this Apache Derby in-memory cache, we can often intuitively understand it through a heapdump relatively quickly as in this case where there is a class named CacheEntry.

At this point, we can guess that there's some sort of Java in-memory cache for Apache Derby which is likely limited to a size of 1,000. Next, we can consult Apache documentation and/or its development team to understand if there's a way to increase the size of the cache and test the benefit.

An orthogonal question is what is the cache hit ratio which is the proportion of successful attempts of getting something from a cache. This is generally monitored through some sort of monitoring tool or statistics and the mechanism to do so will depend on each cache. Ideally, you want the cache hit ratio to approach 100%.

If the cache object did not jump out in the dump, then it may be that a cache is under-utilized for some reason. Consult your application architects to understand which caches should have been used and then find them by class name in the dump to confirm.

Conclusion

In conclusion, Java in-memory caches are a critical aspect of optimizing the performance of your application. You may use Java heapdumps and the free Eclipse Memory Analyzer Tool to sample the occupancy of various caches and determine whether there are opportunities to test increasing cache sizes or investigating why caches are not being utilized.

#app-platform-swat
#websphere#WebSphereApplicationServer#Liberty#performance#Java​​​​​

Permalink