• Problem scenario:  

In IBM Security identity manager (ISIM) Virtual Appliance cluster environments, sometimes adding a member to the Primary node might fail due to various reasons.  Reasons like:   

1. The time is not synchronized with the primary node.
2. Primary node is down or not reachable from the member node.
3. 3. The member node is not configured with fully qualified hostname.
4. 4. Too many attempts to add the
5. 5. Member node configuration in hung state.
6. Primary node is either not configured properly and or not in healthy state.

• Environment: IBM Security Identity Manager Virtual Appliance 7.0.1.* 

• Solution:  

The below link is to add member node to create the cluster environment: 
https://www.ibm.com/support/knowledgecenter/en/SSRMWJ_7.0.1.11/com.ibm.isim.doc/installing/tsk/t_using_cluster_config_wizard.htm 

Steps are easy to add member node to the primary node, but it might fail sometimes. There are few points to consider to either avoid failures or resolve them.  

Precautions: 

1. Make sure that the member node date and time are same as that of the primary node of virtual appliance.
2. Before configuring member node, make sure that it is in same environment where it can connect properly to Primary node. Do not configure firewall between those nodes where it can block some connections. 
3. 3. In some cases where local host file entries or DNS settings are not proper or incomplete, it is required to have the member node with fully qualified hostname. Short host name of the member node might result into failures.

Error handling: 

1. 1. If an end user tries to configure member node even before the primary node is not configured properly and in healthy state then there are chances of failures to add member to it. In such cases, first let primary node configuration get completed where all middle ware like LDAP server, DB, Mail server and application interface is proper and configured. If any of the middle-ware is not configured properly there might be an error like below while adding member node:

"The Primary node was not configured correctly." error occurred. 

2. 2. There can be case where end user tries to add member node multiple times and have failures. The incomplete setup of member node might leave some residue on Primary node every time and requires proper clean-up for the same. Sometimes cleaning SIB tables and transaction logs on the primary node can help resolving the issue. Refer the below info centre link to clear the SIB tables and transaction logs:

http://www-01.ibm.com/support/docview.wss?uid=swg21380384 

If cleaning SIB tables and transaction log does help then it might require contacting IBM support to get fixpack to clean-up the residue a at primary node. 

3. 3. Sometimes member node configuration goes into hung state due to network or system issues. Check the logs to find if any failure or it can help to determine if the node goes in hung state. Example:

ADMU0116I: Tool information is being logged in file 

/opt/IBM/WebSphere/AppServer/profiles/isimNode8/logs/ISIMVa_APP_MEMBER_8/startServer.log 

ADMU0128I: Starting tool with the isimNode8 profile 

ADMU3100I: Reading configuration for server: ISIMVa_APP_MEMBER_8 

ADMU3200I: Server launched. Waiting for initialization status. 

If the execution does not go further, it might be into hung state. In that case, end user might try killing a java process on member node and then let continue the configuration: 
isim> nodes_administration stop -force 

• Conclusion: 

Before creating cluster environment, always first check the state of primary node and its connectivity from the member node. With proper configurations and steps, a member node add will be successful but in case if it always fails for some reason, check logs to find the root cause. Cleaning residue on primary node, use of new IP or FQDN for member node can also help creating member node.