WebSphere Application Server & Liberty

WebSphere Application Server & Liberty

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Back to discussions
Expand all | Collapse all

OutOfMemory puzzle

  • 1.  OutOfMemory puzzle

    Posted Mon November 16, 2020 06:46 AM
    Hi,

    I cannot figure out why the OutOfMemory was thrown. 
    Looking through the FFDC and native_stderr.log, I cannot see anything, lots of freebytes available.
    The only thing I can spot is, gc type="global", rather than scavenger.
    Curiously, only a third of the Max heap size is being used.

    Any ideas?

    WebSphere 7   Java 1.6

    Initial Heap size 3072Mb

    Max heap size 8192

     

    FFDC logs

    [11/16/20 2:44:49:521 EST]     FFDC Exception:java.lang.OutOfMemoryError

    java.lang.OutOfMemoryError: native memory exhausted

     

     

    native_stderr.log corresponding error

     

     <sys id="3" timestamp="Nov 16 02:44:37 2020" intervalms="241624867.691">

      <time exclusiveaccessms="0.166" meanexclusiveaccessms="0.079" threads="9" lastthreadtid="0x000000000097FA00" />

      <refs soft="79838" weak="119073" phantom="4314" dynamicSoftReferenceThreshold="20" maxSoftReferenceThreshold="32" />

      <nursery freebytes="510871832" totalbytes="734688256" percent="69" />

      <tenured freebytes="1572462664" totalbytes="2415919104" percent="65" >

        <soa freebytes="1451667528" totalbytes="2295123968" percent="63" />

        <loa freebytes="120795136" totalbytes="120795136" percent="100" />

      </tenured>

     

      <pending-finalizers finalizable="145" reference="32582" classloader="0" />

      <gc type="global" id="4" totalid="434" intervalms="30103761.315">

        <compaction movecount="13262577" movebytes="731892912" reason="compact on aggressive collection" />

        <classunloading classloaders="918" classes="603" timevmquiescems="0.000" timetakenms="316.929" />

        <finalization objectsqueued="145" />

        <timesms mark="235.116" sweep="3.375" compact="3287.823" total="3888.070" />

        <nursery freebytes="728656368" totalbytes="734688256" percent="99" />

        <tenured freebytes="1690004512" totalbytes="2415919104" percent="69" >

          <soa freebytes="1569209376" totalbytes="2295123968" percent="68" />

          <loa freebytes="120795136" totalbytes="120795136" percent="100" />

        </tenured>

      </gc>

     

      <nursery freebytes="728656368" totalbytes="734688256" percent="99" />

      <tenured freebytes="1690004512" totalbytes="2415919104" percent="69" >

        <soa freebytes="1569209376" totalbytes="2295123968" percent="68" />

        <loa freebytes="120795136" totalbytes="120795136" percent="100" />

      </tenured>

      <refs soft="25619" weak="110535" phantom="4200" dynamicSoftReferenceThreshold="22" maxSoftReferenceThreshold="32" />

      <pending-finalizers finalizable="145" reference="32582" classloader="0" />

      <time totalms="3888.412" />

    </sys>

     

    JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2020/11/16 02:44:42 - please wait.

     

     

    An hour later, looking at the Admin console > Performance Viewer > Used heap size is only 1.3Gb, the HeapSize is 3Gb, as per  (Initial Heap size 3072Mb)


    Very strange!


    Thanks,

    Mark.



  • 2.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 08:03 AM
    Native memory, is a memory region outside the heap region.

    This means you can have a lot of free heap, but you can eat the whole RAM of yout phisical and virtual machine.

    Native memory leak can be generated for different reason, but sometimes there are application like "file transfert application" that can use this memory for their needs becaus the nio library use  native memory to buffering files and other stuff.

    You should try to increase the memory and you  need to take a core dump and open with the eclipse MAT.

    Take a look to this https://publib.boulder.ibm.com/httpserv/cookbook/Major_Tools-IBM_Memory_Analyzer_Tool.html


    ------------------------------
    Michele Buccarello
    ------------------------------

    Original Message


  • 3.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 09:03 AM
    Any chance you have a 32-bit runtime?  It would not leave much virtual address space for off-heap allocations.

    ------------------------------
    Eric Covener
    ------------------------------

    Original Message


  • 4.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 09:45 AM
    Look at the first line of the generated javacore.  It will say if it is heap exhaustion (not likely) or a native code issue.  If it is a native memory issue, there will be an error number which will provide a clue as to what the cause is.

    ------------------------------
    Dennis Riddlemoser
    Technical Account Manager
    IBM
    ------------------------------

    Original Message


  • 5.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 10:38 AM
    It seems a native 

    FFDC logs

    [11/16/20 2:44:49:521 EST]     FFDC Exception:java.lang.OutOfMemoryError

    java.lang.OutOfMemoryError: native memory exhausted

     I think after he check if he have 32 or 64 bit jvm he should also share some log files.



    ------------------------------
    Michele Buccarello
    ------------------------------

    Original Message


  • 6.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 10:47 AM
    What version of Java is this?  If that the following does not look like Java 8 or even 7.  Looks like it may have been Java 6 or prior.  The Javacore has information on native memory segments.  Use Thread Monitor and Dump Analyzer to look at them or look at them directly in the javacore.  Also, look at the NO_FILE ulimit.  Normally if this is exhausted, an I/O exception is thrown but if it is exhausted allocating a semaphore, a native OOM can be thrown.  The ulimit should be significantly larger than the 1000 default value. 

    java.lang.OutOfMemoryError: native memory exhausted


    ------------------------------
    Dennis Riddlemoser
    Technical Account Manager
    IBM
    ------------------------------

    Original Message


  • 7.  RE: OutOfMemory puzzle

    Posted Mon November 16, 2020 10:42 AM
    Hi,       thanks for the replies, here's an update

    The Linux VM has around 12Gb free.
    It's 64 bit.

    Javacore log, from the top 



    0SECTION       TITLE subcomponent dump routine
    NULL           ===============================
    1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
    1TIDATETIME    Date:                 2020/11/16 at 04:29:43


    Having Googled 'Dump Event "systhrow" (00040000) Detail' it's talking about increasing the ulimit, but that might be a red herring.

    Thanks!



    ------------------------------
    Mark Shaw
    ------------------------------

    Original Message


  • 8.  RE: OutOfMemory puzzle
    Best Answer

    Posted Tue November 17, 2020 02:29 AM
    Hello Mark,
    as this is a 64Bit Linux but you are starting with 3072Mb heap you might I'd suggest to set the preferred heap base to 4GB by adding the following generic JVM argument: 
    -Xgc:preferredHeapBase=0x100000000 ​
    Please see https://www.ibm.com/support/pages/node/507367 for more details.

    Hope that helps - Hermann

    ------------------------------
    Hermann Huebler
    2innovate IT Consulting GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------

    Original Message


  • 9.  RE: OutOfMemory puzzle

    Posted Tue November 17, 2020 05:44 AM
    Hi,

    Indeed you should check the ulimit for your instances users (if you can share with us the limits file).

    if this is a recurring event, i would advise to enable verbose GC and analyze how your memory is used by the application it may give you hints about whether you should increase the heap size or fix a bug causing memory leak in the application.

    Good Luck

    ------------------------------
    Largou walid
    ------------------------------

    Original Message


  • 10.  RE: OutOfMemory puzzle

    Posted Tue November 17, 2020 01:46 PM
    > Having Googled 'Dump Event "systhrow" (00040000) Detail' it's talking about increasing the ulimit, but that might be a red herring.

    Not necessarily a red herring.  Native OOM could mean that you have too many file handles open - often a javacore or system dump file can help you determine that.  If that is the case, then it might be necessary to either increase the ulimit for file handles or ensure that the application is closing file references (or both).

    Another possibility for native OOMs are that you have too many threads running.  A javacore/thread dump will tell you that.  If your application creates threads, that may be something to look at - consider using a thread pool.  Or it's possibly that your server may have too many threads configured.  If it looks like there are too many threads in the server JVM compared to the ulimit's threads-per-process value, you may either need to reduce thread usage or increase the ulimit value.

    Hope this helps!

    ------------------------------
    Andy McCright
    ------------------------------

    Original Message


  • 11.  RE: OutOfMemory puzzle

    Posted Tue November 17, 2020 10:26 AM
    Hi Mark, As others have noted, this is a native OutOfMemoryError. This is either due to exhausting virtual memory within 0-4GB with compressed references enabled, or ulimit issues. Normally, it's the former. Such issues are not related to how much RAM you have available; it's a virtual memory issue. For compressed references to work, native structures backing Java classloaders, threads, and monitors must be allocated in the 0-4GB native virtual native memory range. Review the NATIVEMEMINFO section in the javacore to see if something is consuming many GB of virtual memory. The most common cause is a classloader leak or DirectByteBuffers. In such cases, review the core dump to resolve the leak or excessive usage driving native memory. As Hermann mentioned, you should also check the virtual address location of your Java heap and move it up with preferredHeapBase if it's below 4GB virtual. For details, see https://publib.boulder.ibm.com/httpserv/cookbook/Troubleshooting-Troubleshooting_Java-Troubleshooting_OpenJ9_and_IBM_J9_JVMs.html#Troubleshooting-Troubleshooting_OpenJ9_and_IBM_J9_JVMs-Native_OutOfMemoryErrors_on_64bit

    ------------------------------
    Kevin Grigorenko
    App Platform SWAT
    IBM
    ------------------------------

    Original Message


  • 12.  RE: OutOfMemory puzzle

    Posted Wed November 18, 2020 09:10 AM
    In the javacore that reports "native memory exhausted" have a look at the thread stack for the "Current thread". If it shows the current thread was trying to launch a new thread or load a class at the time of the native OutOfMemoryError, then it's likely to be the issue with exhausting the 0 - 4GB memory range while using the JVM compressed references mode. Alternatively the current thread stack might show a call to a native method that is trying (and failing) to get a memory allocation from the OS.

    ------------------------------
    Will SMITH
    ------------------------------

    Original Message


  • 13.  RE: OutOfMemory puzzle

    Posted Fri December 11, 2020 11:11 AM
    Hi,

    Just to report back the good news. After increasing the JVM Initial Heap size from 3Gb to 4Gb that has resolved the issue, as per :
    https://www.ibm.com/support/pages/node/507367 

    A big thanks to everyone who took the time to answer.

    ------------------------------
    Mark Shaw
    ------------------------------

    Original Message


Related Content