Heap Dump Problem WebSphere Application Server 6.0

--------------------------------

Hi Friends

Initial Heap Size 512 MB
Maximum Heap Size : 4 GB
System Ram Size : 8 GB

Heap dump files are generating 2 days once in production server . we are not able to trace the root cause ,we dont know wheather our program problem or application server settings problem

we are getting OutOfMemory in SysErr.log but we dont find any error or exceptions in SystemOut.log

Error Msg :
[code]
[5/27/09 10:32:03:341 GST] 00000421 SystemErr R at com.ibm.ws.util.WSThreadLocal.getThreadLocals(WSThreadLocal.java(Inlined Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.util.WSThreadLocal.get(WSThreadLocal.java(Inlined Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.util.objectpool.TwoTierObjectPool.getThreadLocalObjectPool(TwoTierObjectPool.java(Inlined Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.util.objectpool.TwoTierObjectPool.get(TwoTierObjectPool.java(Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.buffermgmt.impl.WsByteBufferPool.getEntry(WsByteBufferPool.java(Inlined Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateCommon(WsByteBufferPoolManagerImpl.java(Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateDirect(WsByteBufferPoolManagerImpl.java(Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.tcp.channel.impl.SocketIOChannel.attemptReadFromSocket(SocketIOChannel.java(Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
[5/27/09 10:32:03:357 GST] 00000421 SystemErr R at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
[5/27/09 10:32:05:685 GST] 00000421 SystemErr R at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
[5/27/09 10:32:05:685 GST] 00000421 SystemErr R at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
[5/27/09 10:32:53:933 GST] 00000422 SystemErr R java.lang.OutOfMemoryError
at com.ibm.ws.util.WSThreadLocal.getThreadLocals(WSThreadLocal.java(Inlined Compiled Code))
at com.ibm.ws.util.WSThreadLocal.get(WSThreadLocal.java(Inlined Compiled Code))
at com.ibm.ws.util.objectpool.TwoTierObjectPool.getThreadLocalObjectPool(TwoTierObjectPool.java(Inlined Compiled Code))
at com.ibm.ws.util.objectpool.TwoTierObjectPool.get(TwoTierObjectPool.java(Compiled Code))
at com.ibm.ws.buffermgmt.impl.WsByteBufferPool.getEntry(WsByteBufferPool.java(Inlined Compiled Code))
at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateCommon(WsByteBufferPoolManagerImpl.java(Compiled Code))
at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateDirect(WsByteBufferPoolManagerImpl.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.SocketIOChannel.attemptReadFromSocket(SocketIOChannel.java(Compiled Code))
at com.ibm.ws

Chances are that you have a memory leak. Are you getting a heapdump? If so, I would recommned downloading HeapAnalyzer to help identify the leak.

www.alphaworks.ibm.com/tech/heapanalyzer

--------------------------------

Posted By: cathy at Jun 1 2009 3:28PM

Hi Arun

What version of WebSphere are using and which OS?

I'm afraid to say but 9 times out of 10 memory leak is due to user code. :(

We had a number of memory leaks in our applications that the developers said could never be them.

Obvious areas of investigation would be looking at any static references within the code (hopefully none) one example we had was a static cache (a hash map) then ended up growing to 2Gb in size causing an out of memory.

Other examples of growing memory that we've come across is looping through resultsets that are too big. Reading CLOBs or BLOBs from a database that were massive. Circular referencing of objects so they never get destroyed by the GC. Not closing input streams or giving back database connections.

Also check your connection cache size settings I've had an example in the past where we set the statement cache size too large because we saw a high number of cache discards and try to over come this by increasing this but then reduced the heap size available.

To aid investigation switch on verbose GC and download ga140.jar use this to monitor the growth of the heap size. You want to look at Mark, Sweep, Compact and time since last GC.

Hope this helps.

Frankie

--------------------------------

Posted By: frankstar123 at Jun 2 2009 10:50AM

RE: Heap Dump Problem WebSphere Application Server 6.0

--------------------------------

thanks for your reply

I download the IBM Heap dump anlayzer tool(ha37).
Whenever the i gave the latest heap dump file, each time i got the different suspected objects such as Transaction object, Connection Pool object, XML Bean object.
i dont think ,there will be problem in application level.

--------------------------------

Posted By: arunssoft at Jun 2 2009 8:26AM

RE : Heap Dump Problem WebSphere Application Server 6.0

--------------------------------

Hi friend

OS : Windows Server 2003
App Server : WebSphere Application Server 6.0

thanks for ur reply and concrete examples.
we didnt get any heap dump file for the past 2 years . we are facing the problem since last 3 to 4 months . number of cuncurrent online users 250 to 300 users , now our primary table having above 350000 records.

we have around 60 stateless beans with single hashmap argument and 200 jsp pages with one servlet (controller)
we are using cachedRowSet for getting results (only for select operations) .with fetching limited rows (max 100) and 10 final static variables (maximum it ill take 7MB data)

--------------------------------

Posted By: arunssoft at Jun 3 2009 7:24AM

Ok it seems like quite a controlled application.

The next line of investigation i'd do is switch on verbose gc (-verbose:gc on cmd line)

download the following tool and monitor the native_stderr.log for a given app server.
www.alphaworks.ibm.com/tech/pmat

Hopefully the graphs produced may let you know if its a single point in time where there's big ramp up of memory consumption or if its continuous growth.

--------------------------------

Posted By: frankstar123 at Jun 8 2009 9:51AM

With SDK 1.4.2, you may also want to check the verbosegc to see if you might have an issue with fragmentation. You may have room in the heap, but it may be fragmented so allocation requests cannot be fulfilled.

--------------------------------

Posted By: cathy at Jun 11 2009 1:57AM

Heap Dump Problem WebSphere Application Server 6.0

--------------------------------

Hi friends

is there any difference between websphere server verbose gc and java verbose gc , i think both are same because websphere app server 6.0 having own ibm java 1.4.2

thanks for your reply

Regards
Arunkumar S

--------------------------------

Posted By: arunssoft at Jun 11 2009 7:12AM

From a JVM perspective, WSAS is simply a java program. The JVM produces the verbosegc. WSAS 6.0 runs on 1.4.2 SDK.

So yes - they are one and the same.

--------------------------------

Posted By: cathy at Jun 15 2009 1:49AM

Heap Dump Problem WebSphere Application Server 6.0

--------------------------------

hi friends

i got native_stderr and native_stdout files when Generating a heap file .but there is only out of memory error information, no other information for analyze

native_stderr.log
-----------------
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
JVMDG318: Heap dump file written to C://IBM//WebSphere//AppServer//profiles//default//heapdump.20090411.121853.7196.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to C://IBM//WebSphere//AppServer//profiles//default//javacore.20090411.121858.7196.txt
JVMDG274: Dump Handler has Processed OutOfMemory.
JVMST109: Insufficient space in Javaheap to satisfy allocation request
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
JVMDG318: Heap dump file written to C://IBM//WebSphere//AppServer//profiles//default//heapdump.20090411.121918.7196.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to C://IBM//WebSphere//AppServer//profiles//default//javacore.20090411.121923.7196.txt
JVMDG274: Dump Handler has Processed OutOfMemory.
etc..
---------------------------

native_stdout.log (only dots(.) )
--------------------------
...............................................................................
................................................................................

--------------------------------

Posted By: arunssoft at Jun 24 2009 5:12AM

Do you have verbosegc enabled? If so, you should get information in native_stderr.log concering gc.

You can also check the javacore and see how big the heap is and how many bytes are free ---- helps to determine if fragmentation is a problem.

Also, you have the heapdump you can run thru the heap analyzer tool

--------------------------------

Posted By: cathy at Jun 25 2009 12:33AM