Originally posted by: SystemAdmin

I've a problem with a hacmp. This morning 8 machines (4 production machines and 4 ha) rebooted. The logs said that Heartbeats were missed.
It seems as a network problem, but I would like to know if may be a CPU usage issue.
02/20 09:15:02:hatsd[2]: Missed Heartbeats = 2.
02/20 09:15:04:hatsd[0]: Missed Heartbeats = 3.
02/20 09:15:10:hatsd[0]: Missed Heartbeats = 4.
02/20 09:15:10:hatsd[0]: Leader (172.16.96.222:0x85d81d7d) died, notifying crown prince (172.16.96.221:0x85d820
94). FFDC id http://.8hjsyyCseq3/raJ.R1K2gy....................
02/20 09:15:10:hatsd[0]: Received a DEATH IN FAMILY message from (172.16.96.221:0x85d82094) in group (172.16.96
222:0x85d820bc).
02/20 09:15:10:hatsd[0]: Leader died, assuming the responsibilities of the Leader.

Then there is the dead man message in errpt:

LABEL: SYSLOG
IDENTIFIER: C6ACA566

Date/Time: Tue Feb 20 09:14:38
Sequence Number: 28857817
Machine Id: 0053990A4C00
Node Id: XE0001
Class: S
Type: UNKN
Resource Name: syslog

Description
MESSAGE REDIRECTED FROM SYSLOG

User Causes
OPERATOR REDIRECTED SYSLOG MESSAGES TO ERROR LOG

Recommended Actions
REVIEW DETAILED DATA

Detail Data
SYSLOG MESSAGE
<130>Feb 20 09:14:38 clstrmgr10994: Tue Feb 20 09:14:38 AcceptDeadman: Accept failed for socket 1

and after this, there is an stop in the server:

user@machine /home/user>errpt|grep -i shut|pg
2BFA76F6 0220091407 T S SYSPROC SYSTEM SHUTDOWN BY USER

Any body knows where the problem may be?
#AIX-Forum

Originally posted by: orphy

It's difficult to say exactly what caused the deadman from the info given
but it sounded like you are only using hb over TCP/IP. I'm not sure if you
have a 8-node clstr or not but it also sounded like it that way. Assuming
that you only hb over IP, having all 8 nodes going down usually means you have
a big issue at the network level. It's hard for me to believe that all 8 nodes
(both primaries and standbys) are having such a high CPU load that would miss
all those hbs.

Having such a good size cluster, I would assume that you have a support contract
with IBM. If so, I'd recommend opening a PMR with them. They will be able to
look at the snap and tell you the cause of the deadmans.

If you only have one hb path, I suggest that you investigate and add some other
ways of doing hbing. See the Concepts Guide for more info.
Orphy
#AIX-Forum

Originally posted by: SystemAdmin

They are 4 diferent cluster: 1 production machine 1 for High Availability.
They are connected 1 to 1 via tty->micromodem->fiber->micromodem->tty
There are 4 diferent fiber, and I think its dificult to fail all of them in 10 minutes.
2 of production machies have an intensive CPU usage, but the other 2 doesnt.
#AIX-Forum

Originally posted by: orphy

Since they are 4 independent clusters, it had to be something outside those
clstrs that caused the deadmans. I'd be very surprised it was the heavy
CPU on the primaries to kill the boxes especially when the standbys were
mostly idle. If anything, those standbys should have grabbed the resources
when the primaries went down but obviously that didn't happen.

Now, it's also possible that you hit an HA bug! That's why I suggested talking
to IBM so they could look at your clsnaps.

Why do you need to use tty->modem->fiber... type of setup for hb? Are the nodes
of the 2-node clstrs very far apart? I'm still not sure if you are using TCP/IP
hbs. If not, you should consider that since it could have possibly saved the
deadmans.
Orphy
#AIX-Forum