WebSphere Application Server & Liberty

While running a java application, users often encounter memory leak issue. This could be due to Java heap memory leak or native OOM (Out Of Memory) or due to various other reasons. In this blog, we focus on what memory leaks are, what causes those, well known troubleshooting mechanisms for Java Heap Exhaustion and offer recommendations and best practices to follow when running into memory leaks.
 
How does Java manage memory?

Java heap is the memory area used for the allocation and management of java objects created by the application. The process of managing memory in the virtual machine (VM) is handled by the allocator and the garbage collector. The allocator is responsible for assigning areas of the heap for Java objects. The garbage collector reclaims heap space by removing objects when they are no longer required to prevent applications from running out of memory.

When an application creates a new object, it is stored in the java heap. And if the object is no longer needed or goes out-of-scope then that object becomes “garbage” which is eligible for collection by the GC system. GCs are typically triggered by an allocation failure of an object which implies that the free space is exhausted. When a GC occurs, the heap spaces that garbage objects lived in are returned to the heap free memory list, so new objects can be allocated in those memory areas. Even though Garbage collection is doing its job to reclaim memory there are situations where memory leak can still occur.

Why do memory leaks occur?

Memory leak is a situation that occurs when objects that no longer needed are not garbage collected as they still hold references. Garbage collector only cleans unreferenced objects. 

Memory leak may eventually generate Java heap OOM error, if left un-checked. In other words, out-of-memory exception occurs when the live object population requires more space than is available in the Java managed heap. Heap usage depends on the application design, creation of object populations and the interaction between heap and the Garbage Collector. Java heap OOM could be due to Insufficient heap size, Fragmentation or due to memory leaks. Memory leaks are of more concern than memory overuse as it could lead to system instability. 

Memory leaks causes

• Severe Performance degradation
• OutOfMemory Error
• Application crashes 

Consider below code snippet that populates a static List, Note that static fields have lifetime until the end of the program or as long as the class is loaded in memory.

import java.util.*;

public class StaticVariableUse {
	private static List<Integer> nums = new ArrayList<Integer>();
	public void addNums() {
		for (int i = 0; i < 10000000; i++) {
			nums.add(i);
		}
	}
	public static void main(String[] args) throws InterruptedException {
		new StaticVariableUse().addNums();
		System.gc();
		Thread.sleep(100000);
    }
}​

In the above example, after inserting numbers to the list through "addNums" method, the ArrayList is not used in the program. Let us examine what happens to the list object when gc is invoked. To make sure we are seeing the right picture, let us also inject a small delay using Thread.sleep() call. This is to allow the GC to perform a full collection. As garbage collection is automatic and non-deterministic, let us also call it explicitly - using System.gc(), and try to reclaim everything.

We see that static fields are not collected!

Important point to note here however, is that the use of System.gc() is generally not recommended since they can cause long pauses and do not allow the garbage collection algorithms to optimise themselves. It performs stop-the-world major collection and hence not great for applications that desire high throughput. It is used in the above snippet only for illustration purpose.

Debugging Memory Leaks

Troubleshooting memory leaks may require analysis through multiple tools. Let's use couple of most commonly used tools 

1) Using IBM Garbage Collection and Memory Visualizer(GCMV) tool:

Enable your application with Java verbose garbage collection. The data produced from this will help performing performance analysis, debugging OutOfMemoryErrors, and other post-mortem troubleshooting. The GCMV tool can be downloaded from GCMV Latest Release

GCMV provides guidance on fine tuning of Java heap size, Determining peak and average memory usage and recommends Java heap settings.  Enable GC Log Collection for the application and collect the log. Now, let’s load verbose gc log and analyse the data using GCMV. The resulting graph will look like below:

The graph data can be interpreted as follows:

• Used heap is gradually increasing after collection
• GC couldn’t free up much memory even after global gc

This is ratified with the program logic and the contract of the garbage collection – the object “nums” in the program is a static ArrayList, and it has a global access, outside of the life cycle of any Java object.

Now, convert the variable “nums” as a non-static one - remove static keyword from the code snippet at line4, re-run the application with the verbose gc option ON, collect the data and observe the memory usage. The resultant graph is shown as below:

The data from the graph shows that:

• GC is successful in reclaiming the memory
• Used heap after collection decreased drastically from the point highlighted in red.

This is ratified, as the object “nums” that holds large chunk of memory is an instance variable, and has no strong references to it, at the time when the gc runs.

The proportion of time in garbage collection is another important aspect to look at from the GCMV report. Study shows that, in general it should be:

• less than 10% to be called as normally operating,
• and ideally less than 1% to be called as optimally operating

GCMV allows cropping data analysis, customizing views, comparing different data runs, and many more. Additional details about these features and their usages can be referred here - Garbage Collection and Memory Visualizer

2) Using Memory Analyzer Tool(MAT):

Heapdump provides a snapshot of heap memory. The key difference between a heapdump and a system dump (core dump, user dump, mini dump, core) are:

• System dump is created by operating system while heapdump is created by JVM
• System dump is a snapshot of entire address space, whereas heapdump the snapshot of the entire Java heap.

To debug with the help of MAT tool, collect heapdump at points where you suspect there is a memory leak. Best practice suggests to collect system dump itself, as it helps in getting the finest level of details from the application’s memory space. But for most common leak debugging, heapdumps would be sufficient.

Load the dump into the tool. The resulting view (overview tab) is as follows

This overview shows the state of the live heap at the time of dump (MAT performs a full garbage collection when collecting the dump, so this does not include any garbage).

Next, to pick up few leak suspects based on the volume of their occupancy in the heap and your perception about their expected occupancy. For each such suspects, perform the following:

Click on the biggest dominator objects and look for outgoing references. Outgoing references are references to objects that are contained in the current object. That is, if you find A has an outgoing reference to B, that means object A contains object B as a reference.

When you traverse through the outgoing reference chain, you get numerous possibilities to walk – depth-first-search, breadth-first-search or random traversal. Which one will you follow? None of these. You follow the path by looking at the retention size of the list of reference objects and picking up either the largest one, or by your perception about the composition of objects.

Below Outgoing references shows that the class StaticVariable holds a reference to ArrayList which has 10 millions of Integer elements totalling to 205MB of retained heap.

Contents of the data element can be looked from inspector tab as shown below

 
Leaksuspects report is one of the most used one for Java OutOfMemoryErrors which will search for likely causes of a leak in the dump.

MAT offers lots of sophisticated and useful features such as

• heap usage by class,
• heap usage by package,
• Open Query Browser feature that provides object lookup
• and also provides DirectBuffer Details, Class Loader leaks, Hung thread stacks and many more through IBM Extensions for Memory Analyzer plugins

Additional details and examples on how to use these MAT features are available at Eclipse Memory Analyzer Tool

Best Practices/Useful tips

• Tune heap settings to align with the workload characteristics and the gc policy in effect. In few situations it is just that the application needs more memory so increasing the heap might help.
• In general restarting the application temporarily mitigates the issue, but root cause analysis should be done and resolution applied, for chieving high resiliency.
• Understand the gc policy in effect. Each gc policy uses its own algorithms to clean the heap. Increasing the heap is not always an ideal solution as it might increase pause times depending on the GC Policies in use.
• Use a heap visualizer tool and heapdump analyzer tool (optionally) to make sure health and stability of your java heap.
• Understand the storage targets of your application and make sure they have a deallocation cycle.
• Understand the memory retention behaviour of collection types that you use and understand their implications.