WebSphere Application Server & Liberty

 View Only
Expand all | Collapse all

DMGR & Node agent is going down automatically after some time

  • 1.  DMGR & Node agent is going down automatically after some time

    Posted Tue March 09, 2021 03:20 PM

    DMGR & Node agent is going down automatically after some time.

    We have installed IBM WAS 9 on Windows Server 2016.

    We have raised Severity 1 ticket with Ref. TS005173108.

    Awaiting your call since 4 hours. Request your call back urgently.



    #Support
    #SupportMigration
    #WebSphereApplicationServer(WAS)


  • 2.  RE: DMGR & Node agent is going down automatically after some time

    Posted Tue March 09, 2021 04:24 PM

    I see the case has since been update.

    We will continue to work with you via the case.

    Thanks



    #Support
    #SupportMigration
    #WebSphereApplicationServer(WAS)


  • 3.  RE: DMGR & Node agent is going down automatically after some time

    Posted Sat January 20, 2024 03:53 PM

    Hai

       Please give suggestion for above issue. I am facing same issues but it's on rhel 7.1

    Th



    ------------------------------
    korlapati mohan krishna
    ------------------------------



  • 4.  RE: DMGR & Node agent is going down automatically after some time

    Posted Mon January 22, 2024 04:48 AM

    Which version of WAS? Could you please post the all the log files in the <PROFILE>/logs .



    ------------------------------
    LI MIN YU
    ------------------------------



  • 5.  RE: DMGR & Node agent is going down automatically after some time

    Posted Tue January 23, 2024 04:19 AM

    Hi

        sorry i am not able upload logs due to bank server.its ibm maximo application is deployed.

    Korlapati mohan Krishna



    ------------------------------
    korlapati mohan krishna
    ------------------------------



  • 6.  RE: DMGR & Node agent is going down automatically after some time

    Posted Tue January 23, 2024 04:58 AM

    Search stopServer.sh to find who stop the server on the OS history shell on RHEL 7.1, and search WSVR0023I and WSVR0024I keyword in SystemOut.log , and Maybe there have some JMX call to stopServer from some monitor software.



    ------------------------------
    LI MIN YU
    ------------------------------



  • 7.  RE: DMGR & Node agent is going down automatically after some time

    Posted Wed January 24, 2024 12:41 AM
      |   view attached
    Hai LIMINYU


          please find below image my jmx properties.


    Thanks,
    Korlapati Mohan krishna.





  • 8.  RE: DMGR & Node agent is going down automatically after some time

    Posted Wed January 24, 2024 10:47 AM

    Please describe your issue : Is DMGR & Node agent going down automatically after some time?

    and provide some info

    1.  Which version of WAS?

    2. search WSVR0023I and WSVR0024I keyword in SystemOut.log

    3. Search stopServer.sh to find who stop the server on the OS history shell on RHEL 7.1

    4. Do you have some JMX call to stopServer from some monitor software from remote server?



    ------------------------------
    LI MIN YU
    ------------------------------



  • 9.  RE: DMGR & Node agent is going down automatically after some time

    Posted Thu January 25, 2024 12:41 AM

    Hai,

       dmgr is not down but nodeagent and JVM are  going down automatically after some time.

    1) WAS version 8.5.5.3

    2)I did not find  WSVR0023I and WSVR0024I keyword in SystemOut.log

    3)Backend JVM and nodeagent are running but in dmgr console nodeagent goes down state. at time i am unable hit application.after some time nodeagent and JVM process id are automatically killed.

    Thanks,

    Korlpati Mohan Krishna.



    ------------------------------
    korlapati mohan krishna
    ------------------------------



  • 10.  RE: DMGR & Node agent is going down automatically after some time

    Posted Thu January 25, 2024 05:26 AM
    1. Check all nodeagent log files in the logs and ffdc directory.
    2. Add minimal JVM heap 256M and Maximum JVM heap 1024M for nodeagent and restart it.


    ------------------------------
    LI MIN YU
    ------------------------------



  • 11.  RE: DMGR & Node agent is going down automatically after some time

    Posted Mon January 29, 2024 05:28 AM
    Hai Liminyu


    We face below issue for nodeagent stopping in ffdc log.

    [1/19/24 9:12:34:614 EAT]     FFDC Exception:org.omg.CORBA.TRANSIENT SourceId:com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl.reactivateServers ProbeId:906 Reporter:com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl@b495fc8e
    org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible  vmcid: IBM  minor code: E07  completed: No
    at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1276)
    at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1461)
    at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1164)
    at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1427)
    at com.ibm.rmi.corba.ClientDelegate.request(ClientDelegate.java:1886)
    at com.ibm.CORBA.iiop.ClientDelegate.request(ClientDelegate.java:1383)
    at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:458)
    at com.ibm.ws.orb.services.lsd._ORB_ServerStub.ping(_ORB_ServerStub.java:34)
    at com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl.reactivateServers(LocationServiceImpl.java:885)
    at com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl.getNextTarget(LocationServiceImpl.java:430)
    at com.ibm.ws.orbimpl.services.lsd.LocationServiceDaemon.getDirectIOR(LocationServiceDaemon.java:203)
    at com.ibm.rmi.iiop.Connection.doLocateRequestWork(Connection.java:3196)
    at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2964)
    at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:64)
    at com.ibm.ws.giop.threadpool.WorkQueueElement.dispatch(WorkQueueElement.java:174)
    at com.ibm.ws.giop.threadpool.PooledThread.handleRequest(PooledThread.java:85)
    at com.ibm.ws.giop.threadpool.PooledThread.run(PooledThread.java:102)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
    Caused by: org.omg.CORBA.COMM_FAILURE: CONNECT_FAILURE_ON_SSL_CLIENT_SOCKET - JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred.  Reason:  Connection refused Remote Host: 10.3.77.53  Remote Port: 36941  vmcid: 0x49421000  minor code: 80  completed: No
    at com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl.tryToCreateConnectedSSLSocket(WSSSLClientSocketFactoryImpl.java:308)
    at com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl.createSSLSocket(WSSSLClientSocketFactoryImpl.java:209)
    at com.ibm.ws.orbimpl.transport.WSSSLTransportConnection.createSocket(WSSSLTransportConnection.java:236)
    at com.ibm.ws.orbimpl.transport.WSSSLTransportConnection.createSocket(WSSSLTransportConnection.java:315)
    at com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.java:359)
    at com.ibm.ws.orbimpl.transport.WSTransport$1.run(WSTransport.java:503)
    at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
    at com.ibm.ws.orbimpl.transport.WSTransport.getConnection(WSTransport.java:500)
    at com.ibm.CORBA.transport.TransportBase.getConnection(TransportBase.java:188)
    at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:100)
    at com.ibm.rmi.iiop.GIOPImpl.getConnection(GIOPImpl.java:134)
    at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:230)
    at com.ibm.rmi.corba.ClientDelegate.locate(ClientDelegate.java:1983)
    at com.ibm.rmi.corba.ClientDelegate._createRequest(ClientDelegate.java:2008)
    at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1186)
    at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1272)
    ... 17 more

    CapturedDataElements begin
    arg0 BEGIN:com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl@b495fc8e
    org.omg.CORBA.portable.ObjectImpl::__delegate BEGIN:com.ibm.CORBA.iiop.ClientDelegate@4affd9
     com.ibm.rmi.corba.ClientDelegate::CLASS:com.ibm.rmi.corba.ClientDelegate
     com.ibm.rmi.corba.ClientDelegate::orb BEGIN:com.ibm.CORBA.iiop.ORB@9a0ce3b9
      org.omg.CORBA.ORB::ORBClassKey:org.omg.CORBA.ORBClass
      org.omg.CORBA.ORB::ORBSingletonClassKey:org.omg.CORBA.ORBSingletonClass
      org.omg.CORBA.ORB::defaultORB:com.ibm.CORBA.iiop.ORB
      org.omg.CORBA.ORB::defaultORBSingleton:com.ibm.rmi.corba.ORBSingleton
       org.omg.CORBA.ORB::singleton:com.ibm.rmi.corba.ORBSingleton@931fcd34 depth limit reached
       com.ibm.rmi.ORB::interop:com.ibm.rmi.util.Interop@ecc0f2b depth limit reached
      com.ibm.rmi.corba.ORB::CLASS:com.ibm.rmi.corba.ORB
      com.ibm.rmi.corba.ORB::LSD_PLUGIN:com.ibm.CORBA.iiop.LSDPlugin
      com.ibm.rmi.corba.ORB::PMI_PLUGIN:com.ibm.CORBA.iiop.PMIPlugin
      com.ibm.rmi.corba.ORB::ASYNCH_PLUGIN:com.ibm.CORBA.iiop.AsynchronousServantPlugin
      com.ibm.rmi.corba.ORB::WLM_PLUGIN:com.ibm.CORBA.iiop.WLMPlugin
      com.ibm.rmi.corba.ORB::J2EE_CONTAINER_PLUGIN:com.ibm.CORBA.iiop.J2EEContainerPlugin
      com.ibm.rmi.corba.ORB::THREADPOOL_PLUGIN:com.ibm.CORBA.iiop.ThreadPool
      com.ibm.rmi.corba.ORB::TRANSPORT_PLUGIN:com.ibm.CORBA.transport.Transport
      com.ibm.rmi.corba.ORB::READERPOOL_PLUGIN:com.ibm.CORBA.transport.ReaderPool
      com.ibm.rmi.corba.ORB::IIOPCHANNEL_PLUGIN:com.ibm.CORBA.channel.IIOPChannelPlugin
      com.ibm.rmi.corba.ORB::GIOPCHANNEL_PLUGIN:com.ibm.CORBA.channel.giop.GIOPChannelPlugin
       com.ibm.rmi.corba.ORB::FIRST_PREREQ_PLUGINS:[Ljava.lang.String;@5308203d depth limit reached
       com.ibm.rmi.corba.ORB::SECOND_PREREQ_PLUGINS:[Ljava.lang.String;@7d6d9fcd depth limit reached
       com.ibm.rmi.corba.ORB::_dynamicRequests:java.util.Vector@21ec337f depth limit reached
       com.ibm.rmi.corba.ORB::_svResponseReceived:com.ibm.rmi.corba.SynchVariable@9d9e6eb9 depth limit reached
      com.ibm.rmi.corba.ORB::ORBInitialHost:icdwas.cbe.com.et
      com.ibm.rmi.corba.ORB::ORBInitialPort:2909
      com.ibm.rmi.corba.ORB::ORBServerHost:icdwas.cbe.com.et
      com.ibm.rmi.corba.ORB::ORBServerPort:9900
      com.ibm.rmi.corba.ORB::appletHost:null
      com.ibm.rmi.corba.ORB::appletCodeBase:null
      com.ibm.rmi.corba.ORB::charEncoding:83951617
      com.ibm.rmi.corba.ORB::wcharEncoding:65792
      com.ibm.rmi.corba.ORB::wcharDefault:0
      com.ibm.rmi.corba.ORB::serverId:298002686
      com.ibm.rmi.corba.ORB::orbId:null
      com.ibm.rmi.corba.ORB::noProprietaryActivation:false
       com.ibm.rmi.corba.ORB::arguments:[Ljava.lang.String;@2806777d depth limit reached
       com.ibm.rmi.corba.ORB::initProps:java.util.Properties@5baf4850 depth limit reached
      com.ibm.rmi.corba.ORB::initApplet:null
      com.ibm.rmi.corba.ORB::bufferSize:2048
      com.ibm.rmi.corba.ORB::fragmentSize:1024


    Thanks,
    Korlapati Mohan Krishna.






  • 12.  RE: DMGR & Node agent is going down automatically after some time

    Posted 11 days ago

    Hi Mohan,

    We are also facing the same issue in the recent times. The node agents and the Servers are automatically shutting down and logs are clear.

    Could you please help on how to resolve the issue if you have any solution regarding this.

    The below are some thing logged in the FFDC Folder.

    CORBA.COMM_FAILURE: CONNECT_FAILURE_ON_SSL_CLIENT_SOCKET - JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred.  Reason:  Connection refused: connect Remote Host: xxxxx Remote Port: 9408  vmcid: 0x49421000  minor code: 80  completed: No

     org.omg.CORBA.COMM_FAILURE: CONNECT_FAILURE_ON_SSL_CLIENT_SOCKET - JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred.  Reason:  Connection refused: connect Remote Host: xxxxx Remote Port: 9408  vmcid: 0x49421000  minor code: 80  completed: No

    org.omg.CORBA.OBJECT_NOT_EXIST: SERVANT_NOT_FOUND (2) for 0x4a4d424900000010621f4a740000000000000000000000000000000000000024000000080000000000000000  vmcid: IBM  minor code: C12  completed: No

    Thank you.



    ------------------------------
    Chaitanya Kumar R
    ------------------------------



  • 13.  RE: DMGR & Node agent is going down automatically after some time

    Posted 10 days ago

    Hai Chaitanya Kumar R,

             I am also facing same issue two months back still i am facing issue not resolved it is my personal system. please raise pmr.you  get any solve please post it.

    Thanks & Regards,

    Korlapati Mohan Kriishna.



    ------------------------------
    korlapati mohan krishna
    ------------------------------



  • 14.  RE: DMGR & Node agent is going down automatically after some time

    IBM Champion
    Posted 9 days ago

    Hello Chaitanya,

    possibly the dmgr renewed its certificates and could not sync with the nodes?

    Did you try to stop all the servers on the node (stop the nodeagent as well), then run a `syncNode.sh` from the command line and then start the nodeagent and check if the nodes connects and can sync the repository? 



    ------------------------------
    Hermann Huebler
    Alpium IT Solutions GmbH
    Vienna
    Austria

    #IBMChampion
    ------------------------------



  • 15.  RE: DMGR & Node agent is going down automatically after some time

    Posted 10 days ago

    Chaitanya,
    Those are ORB-related exceptions which handles various communications between WebSphere servers.  However the ORB itself does not start/stop servers automatically.  I suggest you open a case so that IBM support can review all the logs involved and determine the root cause of your issues.

    Claudia Barrett (IBM ORB L3 Support)



    ------------------------------
    Claudia Barrett
    ------------------------------