Managed File Transfer

This blog describes the method for securing  connections between pairs of Apache Zookeeper (ZK) nodes and  connections between IBM Sterling Global Mailbox (GM)  and ZK nodes. Generally, it is expected that each ZK and GM node has its own key-certificate; the public certificate of which is given to other nodes, who want to connect securely. Generating, securing, maintaining, and renewing the large number of such certificates is operationally intensive for organizations, especially when they can afford limited downtime due to business considerations. The below implementation steps are for the enterprise scenario, wherein all the applications within a data centre use a common SSL certificate, which is similar or different from the certificate used by other data centre.

The blog assumes that reader has basic knowledge of GM implementation, operation, and components. For reference, read the IBM official documentation on this topic.

For illustration, we considered the this topology:

The below mentioned suggestions and steps are tested with IBM Sterling Global Mailbox version 6.0.3.2 hotfix 2 and supported till version 6.1.

Pre-configuration tasks:

1. Type of certificate: To support a common certificate across all the servers in a data centre, Subject Alternative Name (SAN) certificates are needed, which are certificates that have a common name which is not the hostname. This common certificate specifies all the hostnames in the SAN and thus is a multi-domain SSL certificate.
   
   
2. Keystores and trust stores: 
   The certificates are converted to Java Keystore (JKS) format. All the certificates and stores created are kept at same location in each of the servers. These stores are referred in configurations where SSL needs to be enabled by using custom store location.
   
   

3. If we want to ensure zero downtime during certificate renewal, – the file should get processed from one data centre, while the certificate renewal occurs in the other data centre. Then each data centre should have a different certificate and during renewal of certificates in first datacenter (DC1), loadbalancers should be configured to route traffic to the other datacenter (DC2). Secondly, leaf certificates should be removed from all trust stores because root and intermediate certificate(s) are sufficient to validate current as well as future key certificates. The underlying assumption to implement zero downtime is that all key certificates are signed by the same entity (in-house or third party) and thus root and intermediate remain same in foreseeable future. Therefore, to apply new certificates as part of certificate renewal activity, the implementer only needs to replace the current keystore with the new keystore (having the new certificate), restart the applications and perform validation by doing basic smoke testing to ensure file transfer and replication.
   
   
4. Before doing setups in any specific section, please shutdown watchdog (one of the pre-requisites of GM). If watchdog is not shutdown then it will undo the changes in zookeeper configuration files to maintain consistency across nodes. Once all changes are done across nodes then watchdog can be brought up.
   

Now we will see how to setup secured connections from ZK to other components – other ZK nodes, GM and watchdog.

Zookeeper-to-zookeeper encryption

https://www.ibm.com/support/knowledgecenter/SS3JSW_6.0.3/security/security/globalmailbox/gdha_node_to_node_encryption_zookeeper.html

Node-to-node encryption secures the connections between ZooKeeper servers in the cluster with SSL.

We will also need to add :

ssl.quorum.hostnameVerification=false

This disables verification of hostname as common-name in the certificate.

  ‘grep -i 1.2.3.4 zookeeper.log’

B2Bi/GM-to-Zookeeper encryption and authentication

B2Bi/GM acts as a client which tries to establish a secured connection with zookeeper server.

             #clientPort=2181

             secureClientPort=2181

ssl.hostnameVerification=false

The addition of last property disables verification of hostname as common-name in the certificate.

 i. encrypted password for a truststore file by going to the following location on B2Bi and GM node:

<SFG installation path>/MailboxUtilities/admin/bin directory.

Run the following command ./sslEncryptionPasswordUtility.sh password

It generates an encrypted password in the format {xor}your_encrypted_password. For example : {xor}E15MWwABGew==

Copy your encrypted password, inclusive of the {xor} characters, and paste it where required onto your property files. Here ‘xor’ denotes the XOR cipher in cryptography

com.ibm.mailbox.zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty

com.ibm.mailbox.zookeeper.client.secure=true

com.ibm.mailbox.zookeeper.ssl.trustStore.location<path to truststore>

com.ibm.mailbox.zookeeper.ssl.trustStore.password=<encrypted truststore password>

com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

com.ibm.mailbox.zookeeper.ssl.keyStore.location=<path to keystore>

com.ibm.mailbox.zookeeper.ssl.keyStore.password=<encrypted keystore password>

into the following files:

<B2BI install location>/wlp/usr/servers/mailboxui/mailbox.properties,

<B2BI install location>/MailboxUtilities/admin/conf/mailbox.properties

<B2BI install location>/properties/global_mailbox.properties – For this , please update customer_overrides with prefix :

global_mailbox.com.ibm.mailbox.zookeeper.ssl.keyStore.location=<path to keystore>

global_mailbox.com.ibm.mailbox.zookeeper.ssl.keyStore.password=<encrypted keystore password>

global_mailbox.com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

6. Restart the Global Mailbox Client Adapter on every B2Bi node.
 

As mentioned before, we cannot see SSL specific IBM logging in zookeeper node as it is a third-party product. But we can see it on the client side in <B2BI install directory>/wlp/usr/servers/mailboxui/logs/

If ZK connection is not secured then the text will be “Zookeeper Secured Connection is false” else it will be “Zookeeper Secured Connection is true”, “Zookeeper Secured Connection is true. Preparing SSL connection properties.” Some zookeeper configuration properties getting logged can also be seenzookeeper.client.secure, zookeeper.ssl.trustStore.location,

trace.log

messages.log

Enabling Watchdog -ZooKeeper encryption and authentication for ZooKeeper

Watchdog can be configured as mentioned in https://www.ibm.com/support/knowledgecenter/SS3JSW_6.0.3/security/security/globalmailbox/gdha_client(watchdog)_to_node(zookeeper)_encryption_zookeeper.html but with one more addition. GMGM

While configuring encryption and authentication by adding com.ibm.mailbox.zookeeper.ssl.* properties listed below in the <GM Pre-requisite install location>/zookeeper/watchdog/conf/watchdog.properties

also add the property :

com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

By keeping the above mentioned information at hand and implementing the configuration, we achieve secured connections between Apache Zookeeper and other components in an enterprise  level IBM Sterling Global Mailbox setup using multi-domain certificates and ensuring minimal downtime.