Maximo

Maximo

Come for answers, stay for best practices. All we're missing is you.

 View Only

  • 1.  Nodes going down

    Posted Tue April 02, 2024 12:46 PM

    Hi All,

    We have a Maximo 7.6.1 of clustered environment with 3 Nodes. We are seeing all the nodes are shutting down frequently. couldn't get any information from the logs. 

    Can someone help me to solve this issue.



    ------------------------------
    C K K
    ------------------------------


  • 2.  RE: Nodes going down

    Posted Tue April 02, 2024 12:53 PM

    That sounds like you are having an out of memory issue.  If you look in the IBM/WebSphere/AppServer/profiles/ctgAppSrv01/ folder (yours may be slightly different) you will see javacore* files and phd files.  These files have details about the memory usage at the time when the JVM ran out of memory.  Here is the utility to do the analysis: https://www.ibm.com/support/pages/ibm-heapanalyzer

     

    You may also want to check that Verbose Garbage Collection is turned off for the JVMs as one of the WebSphere 9.0.5.x versions introduced Verbose GC as on by default and it absolutely kills performance when it kicks in.  It is great for getting information about garbage collection issues, but for production it should be turned off. You should also check that you have all the various JVM settings, which are documented here: https://www.ibm.com/support/pages/maximo-76-performance-best-practice-guide

     

    Regards,

    Jason

     

     






  • 3.  RE: Nodes going down

    Posted Wed April 03, 2024 03:42 PM
    Hey Jason,

    Thank you for the response. After you described this was an issue with the memory, i went to the server and have the process of the Node in task manager, i observe there was a lot of consumption of memory az well as cpu around 65%. I am not sure on what's the next step.

    Could you please advise on what needs to be done for resolving this?

    Thanks



    Original Message


  • 4.  RE: Nodes going down

    Posted Wed April 03, 2024 02:04 PM

    Hi Chaitanya,

    We've seen clients experience this when they have Sentinel virus software running with 7.6.1.3 clustered so this might be the same for you if you are running Sentinel virus software.  The solution was to add exclusions to Sentinel for the \IBM\WebSphere folders on the servers where WebSphere is installed.  You may have to restart the servers for this to take effect.



    ------------------------------
    Jeff Tippett
    M.G.I. Maintenance Group Inc.
    Halifax NS
    ------------------------------



  • 5.  RE: Nodes going down

    Posted Wed April 03, 2024 03:45 PM
    Hi Jeff,

    Thanks for the response. 

    We are not using the sentinel virus software to run any application or in the server also not sure why the nodes are shutting down. 





    Original Message


  • 6.  RE: Nodes going down

    Posted Thu April 04, 2024 04:19 AM
    Edited by Mark Robbins Thu April 04, 2024 04:21 AM

    As Jason says this sounds like at least one of your JVMs is running out of memory and then having to disconnect from the mesh **. when the JVM disconnects from the mesh there will be messages like the ones that you ask about in your post on the 8th March in the WebSphere forum.

    You say that you have checked the memory usage but I suspect you have been checking that at the operating system level.

    Maximo will grab as much memory as it is allowed and then manage that internally. If a thread asks for memory that isn't available then the dump files that Jason refers to will be created.... at an operating system level it will appear that the JVM is still using the memory that it was and it won't be asking for more than its allowed.

    You need to understand what is using your memory.

    There are two main ways to check this:

    Checking the MBO counts - this will give you an idea about the Maximo Business Objects that are in memory - see this article for an introduction to this https://www.linkedin.com/pulse/seeing-which-mbosetsmbos-using-memory-mark-robbins/

    analysing the heapdump - the JVM would need to be configured to generate heapdumps when the memory is exhausted. It is easy to implement but it can lead to 6GB files being generated on the server and that can cause the overall server to fail because it has run out of disk space

    The whole area is quite complex so I would be surprised if you get a lot of free advice about how to investigate / resolve this.

    Some organisations, like my employer Cohesive, offer support consultancy and we can definitely help you with this.

    I have investigated similar situations in the past so I know that we can help. Message me privately if your organisation would like to buy some consultancy time.

    ** I mentioned the mesh earlier... you may find this article interesting as I talk about the mesh...



    ------------------------------
    Mark Robbins
    Support Lead/Technical Design Authority / IBM Champion 2017 - 2023
    Cohesive (previously Vetasi Limited)
    https://www.linkedin.com/pulse/maximo-support-advice-from-non-ibm-engineer-article-mark-robbins/
    ------------------------------



  • 7.  RE: Nodes going down

    Posted Thu April 04, 2024 04:35 AM

    Did you check the logs in the ffdc folder? inside NodeAgent profile? 
    Is the other JVMs are running correctly?



    ------------------------------
    Sankar Ganesh
    ------------------------------



  • 8.  RE: Nodes going down

    Posted Thu April 04, 2024 07:07 PM

    Hi!

    Here is an oldie but a goodie: https://www.ibm.com/support/pages/maximo-76-performance-best-practice-guide.  This is written by one of the best in the business @Darlene Nerden Pair the info in this doc with what @Jason VenHuizen and @Mark Robbins have said.

    Hope this helps.



    ------------------------------
    Bradley K. Downing , MBA
    Senior Solution Architect
    IBM
    Bakersfield CA
    ------------------------------



Related Content