Hi fellows

I'm setting one Oracle RAC 19c on tow AIX 7.3 TL04 LPARS. As usual one rootvg, on Oracle binVG, shared SAN luns for ASM Voting Disk, Data and ACFS FS. Three LAN interfaces (SRIOV with HNV in order to keep LPM capabilities) one for public access, two on different VLANS for private access and cache fusion. 

Everything looks ok until testing time. When testing failures of any one of the LPARS we found some messages, looks like Oracle is trying to connect (ssh  ???) from the starting node to each of the VIP and SCAN address and then:

PRVG-11067 : TCP connectivity from node "NODE1": "VIP1" to node "bmgacprd16": "SCAN1" failed.
PRVG-11095 : The TCP system call "connect" failed with error "79" while executing exectask on node "NODE1"
Connection refused

And the same message from the same node but changing source IP to any other of the SCAN ip to any other target SCAN IP.

I think is when the starting node tries to open ssh as part as the cluster verification startup sequence but at a given moment this ssh attempt fails. I have some others Oracle RAC running on AIX 7.2 and haven't seen this behavior.

Any ideas ??

TIA

Yes, passwordless SSH back and forth between RAC nodes is a requirement at various times (not during normal run-time if memory serves).  That ought to be documented somewhere in Oracle's support notes.  For Linux deployments, there are pre-check scripts that would flag that explicitly.  I haven't done RAC on AIX in forever, so I'm not sure what's available there.  The passwordless SSH (which -fair warning- historically had some issues with some organizations' standard login warning banners) is used to double-check that configurations match and stuff like that.  If you set up SSH key pairs back and forth, that should go away.  My memory says it's for "oracle" to "oracle", but it might be "grid" to "grid" or both these days.  Sadly, that means opening up one or both for remote login at least temporarily...

Caveats and disclaimers apply...  I haven't done RAC on AIX (or vanilla Linux) in a while...  Good luck!

-Chris

Did you utilize the Oracle cluster verification utility to verify networking, ssh, ...? I find that tool very helpful to verify infrastructure and settings.

I suggest to download the latest version from MOS and run it against your environment.