Hi All,

I have this quesiton bugging me from sometime, tried all things but couldn't find the answer.

When I kill a app server process  as OS level, Nodeagent monitors this and restarts the process again, as we configure it in monitoring policy of app server,

but when we stop the server issuing a stopServer.sh command, nodeagent doesn't start it again.

How nodeagent comes to know if the server was stopped  using command or whether it is killed.


Thanks in Advance.
Pavan

Hi Pavan

with my knowledge

when you enabled the monitor policy for websphere. when the servers gets started under that node. there will be one log monitor.state log will be created. This monitor.state log contains the servername and the process id of them.

when you stop the servers using stopServer.sh cmd the process id will be grace fully removed from OS level as well as from monitor.state log. the nodeagent will stop monitor that stopped server becoz there is not entry for that stopped server in the monitor.state log.

when you kill the process for server , the process id killed manually from the OS and it will not be deleted from monitor.state log. So the nodeagent will ping the server for every 60 sec as mentioned in the monitory policy configuration. and you enable the check box autorestart to start websphere automatically when the nodeagent is running.


Hope this will be useful and correct me If am wrong.

Thanks

monitor.state log will be under logs/nodeagent/monitor.state.

Thanks

Hi AdminUpalla,
Thanks for your prompt response.

I tried removing the entry for a server from monitor.log and  then killed  the  server process at OSLEVEL using kill -9 command.

But still I see nodeagent starting the Server. Is there some other place where it is looking for ?


Thanks,
Pavan

After lot of lookup. I got a convincing answer. It is explained below how the processes are monitored.

The Failure Detection Protocol uses two distinct mechanisms to find failed members:
It looks for connections that closed because the underlying socket was closed.
When a core group member normally stops in response to an administration command, the core
group transport for that member also stops, and the socket that is associated with the transport
closes. If a core group member terminates abnormally, the underlying operating system normally
closes the sockets that the process opened and the socket associated with the core group
transport. is closed.
For either type of termination, core group members that have an open connection to the
terminated member are notified that the connection is no longer usable. The core group member
that receives the socket closed notification considers the terminated member a failed member.
When a failed member is detected because of the socket closing mechanism, one or more of the
following messages are logged in the SystemOut.log file for the surviving members:
DCSV1113W: DCS Stack DefaultCoreGroup at Member anzioCell01\anzioCellManager01\dmgr:
Suspected another member because the outgoing connection to the other member was closed.
Suspected member is anzioCell01\nettuno\ServerB. DCS logical channel is View|Ptp.
DCSV1111W: DCS Stack DefaultCoreGroup at Member anzioCell01\anzioCellManager01\dmgr:
Suspected another member because the outgoing connection from the other member was closed.
Suspected members is anzioCell01\nettuno\ServerB. DCS logical channel is Connected|Ptp.
The closed socket mechanism is the way that failed members are typically discovered. TCP
settings in the underlying operating system, such as FIN_WAIT, affect how quickly socket closing
events are received.

HI Pavan,

Really very nice explanation,

But I have one query, As any of the way force full or gracefull shut down of websphere process, the sockets are closing, the DCS communication(socket/ports) services are closed  to the stopped server.

how the nodeagent restart if the server process is killed forcefully? how it will monitor and restart automatically?

Thanks
Venkata

Hi Venkata,
I am still on my way to find the answer for actual question,  but I am learning quite few new things which I didn't knew. So thought of sharing the same.

I will surely share the details once, I find the answer.


Thanks.
Pavan

HI Pavan

Here is the solution how nodeagent monitors the WebSphere Applicatio Server .

www-01.ibm.com/support/docview.wss?uid=s...

Thanks
Venkata

venkat,
Thanks for the info. Good finding
RK 

Hi Venkata,

Thanks a lot for the link, now it makes sense for me. I have gone through this link earlier, but haven't read the Analysys part, my mistake.

By the way, I am assuming that my earlier post, which expalined how the DCS transport is closed in case of a gracefull shutdown and forceful shut down of the process plays its part in the restart of the server. I came to this conclusion beause I saw DCS messages during this process in nodeagent logs.

Is my perseption correct or is it totally nodeagent that takes care of everything.

I am totally sorry if I am asking silly questions, but I want to learn concepts to perfection.

Regards,
Pavan

HI Pavan

DCS messages are for high availability . These message are logged when there is any cluster member failed or stopped.

as per the Nodeagent monitoring policy the  child processes(servers) will be created under parent(nodeagent) . So the parent is always try to ping the child based on the ping interval whether its up or down. So its full nodeagent responsibility to monitor its  services.

Thanks
Venkata

Hi Venkata,

Thanks a Lot.


Regards,
Pavan