Planning Analytics

Planning Analytics

Get AI-infused integrated business planning

 View Only

  • 1.  TM1 Server OutOfMemory

    Posted Fri January 28, 2022 03:25 AM
    Edited by System Admin Fri January 20, 2023 04:46 PM

    Hello all,

    i got an issue with crashes due to "OutOfMemory". Naturally we thought the maximum RAM was reached, but that isn't the case. It barely reached 55% at the time of the crash:

    The server crashed at the peak with the red arrow.

    We are running the newest TM1 Server version 2.0.9.11 (11.8.010000.6)

    I've checked several settings, like MaximumViewSize (is set to 2GB and no log entry indicates that this was reached).

    Anyone have any idea? Maybe a bug in the TM1 Server version?

    BR,
    Matthias


    ------------------------------
    Matthias Mazaj
    ------------------------------
    #PlanningAnalyticswithWatson


  • 2.  RE: TM1 Server OutOfMemory

    Posted Mon January 31, 2022 02:24 PM
    Hi Matthias,

    I would guess that the system running TM1 actually ran out of memory.  Assuming the memory use of a process spikes very quickly it might not appear on the memory monitoring tool.  Most memory monitoring tools poll the memory utilization of the server on an interval (for example...every 30 seconds).  Maybe the TM1 server process spiked (using all available memory) then crashed within the polling interval.  

    I would start by checking how often the the monitoring tool polls the memory utilization of the server.

    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message


  • 3.  RE: TM1 Server OutOfMemory

    Posted Fri February 04, 2022 03:59 AM
    Hi Stuart,

    thanks for the response. I doubt that the actual limit was reached within a few seconds, as these would mean that the TM1 Server would need to grab almost 55GB RAM. This solution is running for years and the only recent change was the update to a new TM1 Server Version. We never had this issue before but several times after the update.

    We went back to a previous version TM1 Server ( Version 2.0.9.10 (11.8.00900.3) ) and it's running for a few days now without any issues. Again nothing was changes except the TM1 Server version. So I strongly believe there is a fault in 2.0.9.11 (11.8.010000.6).

    BR

    ------------------------------
    Matthias Mazaj
    ------------------------------

    Original Message


  • 4.  RE: TM1 Server OutOfMemory

    Posted Fri February 04, 2022 10:57 AM
    Hi Matthias,

    Without knowing what activity triggered the spike in memory it's impossible to determine root cause.  Assuming a TI process wasn't running at the time of the crash the mostly likely activity to cause this sort of problem is a large MDX view being processes.  If the problem could be reproduce with MDX debug logging enabled that might help us understand the problem.  It would also give us something to test against both 2.0.9.10 and 2.0.9.11.

    https://www.ibm.com/support/pages/how-enable-mdx-debug-logging-tm1-server

    I have not seen a case where the TM1 Server crashes with an out of memory error when there was still available system memory.  Did the error in the tm1server.log file indicate how much memory was used?  Can we see the exact error?

    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message


  • 5.  RE: TM1 Server OutOfMemory

    Posted Mon February 07, 2022 04:25 AM
    Your solutions are always helpful ! Thanks for sharing. vplaynews

    ------------------------------
    Amanda Nunes
    ------------------------------

    Original Message


  • 6.  RE: TM1 Server OutOfMemory

    Posted Wed February 09, 2022 05:03 AM
      |   view attached
    Hi Stuart,

    There was most definitely a TI Process running during the crash or right before. But I doubt that those would use 55 GB of RAM. Especially since the TI processes worked fine before the update and work again now after we installed a previous TM1 Server version. It only crashed in the newest version.

    I've included a part of the tm1server.log right before and after the crash at 11:55

    To provide more details, should I open a support case or message you directly?

    ------------------------------
    Matthias Mazaj
    ------------------------------

    Attachment(s)

    txt
    ServiceFail_tm1serverLOG.txt   89 KB 1 version
    Original Message


  • 7.  RE: TM1 Server OutOfMemory

    Posted Wed February 09, 2022 07:51 AM
    Hi Matthias,

    I don't see anything in the log file that clearly explains the root cause of the problem.  It's not possible to say if a TI process triggered this issue specific to 2.0.9.11.  It does appear that some TI processes failed as the TM1 Server could not open the ODBC proxy process.  This would make sense if the system was out of memort.  We also see a number of out of memory errors being reported by TM1.

    In order for our support team to assist we will need some sort of reproducible testcase.  Do you have a non-production environment with 2.0.9.11 where you can run the same TI processes or attempt to reproduce the problem?  

    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message