IBM webMethods Hybrid Integration

IBM webMethods Hybrid Integration

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

UM shutdown suddenly, wrraper ping jvm

  • 1.  UM shutdown suddenly, wrraper ping jvm

    Posted Wed October 19, 2016 08:04 AM

    Hi community,
    i have a really big problem, i sould deploy some developpement but they don’t want saying that our products aren’t stable, and i need strongly your help,
    The problem is that after every weekend we found our UMs down, when we check our log we found those message :

    2016/10/10 11:03:47 | Shutting down Nirvana Realm
    INFO | jvm 1 | 2016/10/10 11:03:48 | Realm shutdown complete
    INFO | jvm 1 | 2016/10/10 11:03:48 | Nirvana Realm has been shutdown
    STATUS | jvm 1 | 2016/10/10 11:03:48 | Stopping Wrapper monitor on port:9998
    INFO | jvm 1 | 2016/10/10 11:03:48 | [Mon Oct 10 11:03:48 GMT 2016],Server shutdown initiated due to JVM Exit Handler called
    INFO | jvm 1 | 2016/10/10 11:03:49 | [Mon Oct 10 11:03:48 GMT 2016],Shutdown: Status updates shutting down
    STATUS | jvm 1 | 2016/10/10 11:03:49 | Nirvana Realm already shutdown
    STATUS | wrapper | 2016/10/10 11:03:50 | ← Wrapper Stopped
    STATUS | wrapper | 2016/10/10 11:39:51 | Pinging the JVM took 78633 seconds to respond.
    STATUS | wrapper | 2016/10/10 13:27:27 | TERM trapped, but ignored.
    STATUS | wrapper | 2016/10/10 13:34:06 | → Wrapper Sta

    2016/10/15 00:46:45 | Pinging the JVM took 21 seconds

    2016/10/15 03:03:54 | Nirvana Realm Server shutting down due to OutOfMemoryException in Scheduler Worker Pool:4
    INFO | jvm 1 | 2016/10/15 03:05:07 | Nirvana Realm Server shutting down due to OutOfMemoryException in Scheduler
    the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:16:41 | Pinging the JVM took 3 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:17:42 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:18:42 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:21:52 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:25:00 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:29:12 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:31:15 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:33:22 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:34:56 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:38:05 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:41:43 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:42:45 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:45:54 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:49:03 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:50:03 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:54:15 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 06:55:16 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:01:35 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:04:43 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:08:54 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:11:01 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:16:13 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:20:27 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:22:34 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:23:37 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:24:38 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:26:46 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:29:54 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:30:56 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:31:58 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:34:05 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:37:15 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:39:21 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:42:31 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:45:39 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:47:43 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:48:44 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:49:46 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:52:55 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:58:07 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 07:59:09 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:01:17 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:01:49 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:03:23 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:04:25 | Pinging the JVM took 2 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:07:34 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:10:39 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:13:48 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:18:26 | Pinging the JVM took 1 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:26:17 | Pinging the JVM took 3 seconds to respond.
    STATUS | wrapper | 2016/10/19 08:27:17 | Pinging the JVM took 2 seconds to respond.

    it still ping until it shutdown on the weekend,

    we are on webMethods 9,7 AIX, IS_9.7_Core_Fix9,
    i found on the internet this [b]https://wrapper.tanukisoftware.com/doc/english/prop-ping-alert-x.html[/b] but i didn’t understand what’s the relation between wrraper monitoring and UM,

    any suggestions please,

    Regards,
    Nezha


    #Universal-Messaging-Broker
    #webMethods
    #Integration-Server-and-ESB


  • 2.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Wed October 19, 2016 12:41 PM

    Hi Nezha,

    It seems to be issue with Java Heap Memory.

    What is current heap that you have configured for UM?

    Please check and let me know. (location: \SoftwareAG\UniversalMessaging\server\umserver\bin\Server_Common.conf)]

    Also let me know the value of “Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

    Regards,
    Krishna


    #webMethods
    #Universal-Messaging-Broker
    #Integration-Server-and-ESB


  • 3.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Wed October 19, 2016 12:47 PM

    Can you please provide the UM fix level?

    Try to apply the latest fix , hope this will resolve the issue.

    Regards,
    Krishna


    #webMethods
    #Universal-Messaging-Broker
    #Integration-Server-and-ESB


  • 4.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Wed October 19, 2016 02:02 PM

    hi Krishna PB ,
    thank you for your reply,
    the heap jvm configured on Server_Common.conf is 2024,

    about Event Storage → QueueDeliveryPersistentPolicy could you please tell me how to verify this??
    our fixes ares :
    Universal Messaging Java Client 9.7 SP0 Fix 3
    Universal Messaging Enterprise Manager 9.7 SP0 Fix 3
    Universal Messaging Realm Server 9.7 SP0 Fix 3

    Regards,
    Nezha


    #Integration-Server-and-ESB
    #Universal-Messaging-Broker
    #webMethods


  • 5.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Wed October 19, 2016 02:11 PM

    Hi Krishna,
    another setting to check is Event Storage > EnableStoreCaching. It should be set to false.
    You can check/change this setting (and the one metioned by Nezha) in Enterprise Manager. Click on the Realm, select the Config tab and then find the right setting. Double-click it to change the value.

    One other possible cause may be queues or channels without active subscribers. Can you look on the disk in /UniversalMessaging/server//data?
    Are there any very large files in this directory or any of its subdirectories?


    #webMethods
    #Integration-Server-and-ESB
    #Universal-Messaging-Broker


  • 6.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Thu October 20, 2016 04:54 AM

    Hi Jonathan,

    I think you misunderstood , actually Nezha is facing the issue and I suggested to check her.

    Nezha,

    Follow the below instruction to resolve the issue.
    - Try to update the UM fix level to 9.7.0.6/latest. by using update manager
    - Change the setting as I mentioned in my previous post.
    “Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

    To check and change the value for QueueDeliveryPersistentPolicy follow the below steps…

      - login to the Enterprise Manager 
    - Click on  realms 
    - Go to the Config tab
    -Go tot the Event Storage , there you can find the above field and see the value and change it accordingly.
    - Have attached the screen shot for your reference.
    
    • Also check once what Jonathan has mentioned in previous post.

    Please let me know in case of any concerns.

    Regards,
    Krishna


    #Integration-Server-and-ESB
    #webMethods
    #Universal-Messaging-Broker


  • 7.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Thu October 20, 2016 07:13 AM

    hi Krishna and Jonathan ,
    thank you all for your reply,
    @Krishna i will apply the solution that you propose and come back to you for the result,
    i want just to tell @Jonathan that Under this directory /UniversalMessaging/server//data we have very large files which is nirvana logs,

    Regards,
    Nezha


    #Integration-Server-and-ESB
    #webMethods
    #Universal-Messaging-Broker


  • 8.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Thu October 20, 2016 07:46 AM

    hi,
    however on this directory /UniversalMessaging/server//bin there is a lot of very large files dmp & phd & javacore****.txt & .trc,

    Regards,
    Nezha


    #Universal-Messaging-Broker
    #Integration-Server-and-ESB
    #webMethods


  • 9.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Thu October 20, 2016 07:51 AM

    Nezha,
    log files are quite normal and you can safely delete old ones if you want to recover disk space.
    The files in the bin directory look like heap dumps which may have been created automatically by the JVM if it crashed. You can safely delete dmp, phd trc and javacore*.txt files if you don’t think you need them for analyzing some failure.


    #Universal-Messaging-Broker
    #webMethods
    #Integration-Server-and-ESB


  • 10.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Mon October 24, 2016 02:06 PM

    hi,
    could you please tell me which one of thos fixes should i install, see attchement file,

    Regards,


    #webMethods
    #Integration-Server-and-ESB
    #Universal-Messaging-Broker


  • 11.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Mon October 24, 2016 02:27 PM

    Hi Nezha,

    I think you should at least apply Fixes for EnterpriseManager, RealmServer and SPM.

    Remember that all UM Components working with each other should have the same fix level.

    Regards,
    Holger


    #Universal-Messaging-Broker
    #webMethods
    #Integration-Server-and-ESB


  • 12.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Tue October 25, 2016 06:52 AM

    hi everyone,
    Even after installing last fixes and doing all recomended changes, wrraper still pinging the JVM :frowning: see attached files,

    is there anything to do?

    because it beging with pinging jvm and it shuttdown after!!

    we have 3 environnement DEV REC and PROD?

    we havn’t any service or flux runed on REC and PROD environnement, we are in the bigining we still developping on DEV environnement,

    in DEV wrraper ping JVM always(i made changes on this environnement) , on REC it ping sometimes and it shuttdown on weekend, on PROD there is no problem untill now,

    Regards,
    Nezha


    #webMethods
    #Universal-Messaging-Broker
    #Integration-Server-and-ESB


  • 13.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Sat October 29, 2016 08:47 AM

    why they are shutting down only on weekends … Any specific reason ? Did you ask your OS admin team to know any daemon services running which are stopping your UM services … ?

    Thanks,


    #Universal-Messaging-Broker
    #Integration-Server-and-ESB
    #webMethods


  • 14.  RE: UM shutdown suddenly, wrraper ping jvm

    Posted Mon October 31, 2016 01:40 PM

    Hi,

    this might be a license issue.

    When running with specific (invalid) type of license it might force the server to shutdown after a certain period.

    Regards,
    Holger


    #Universal-Messaging-Broker
    #Integration-Server-and-ESB
    #webMethods