IBM Sterling B2B Integrator and IBM Sterling File Gateway Developers

Best practices for securing Apache Zookeeper SSL connections in an enterprise IBM Sterling Global Mailbox setup using multi-domain certificates

By Rahul Jain posted Tue October 20, 2020 01:04 PM

  
           
Co-authored by Girish Venkatesh (girish_venkatest@in.ibm.com


This blog describes the method for securing  connections between pairs of Apache Zookeeper (ZK) nodes and  connections between IBM Sterling Global Mailbox (GM)  and ZK nodes. Generally, it is expected that each ZK and GM node has its own key-certificate; the public certificate of which is given to other nodes, who want to connect securely. Generating, securing, maintaining, and renewing the large number of such certificates is operationally intensive for organizations, especially when they can afford limited downtime due to business considerations. The below implementation steps are for the enterprise scenario, wherein all the applications within a data centre use a common SSL certificate, which is similar or different from the certificate used by other data centre.

The blog assumes that reader has basic knowledge of GM implementation, operation, and components. For reference, read the IBM official documentation on this topic.

For illustration, we considered the this topology:

The below mentioned suggestions and steps are tested with IBM Sterling Global Mailbox version 6.0.3.2 hotfix 2 and supported till version 6.1.

Pre-configuration tasks:

  1. Type of certificate: To support a common certificate across all the servers in a data centre, Subject Alternative Name (SAN) certificates are needed, which are certificates that have a common name which is not the hostname. This common certificate specifies all the hostnames in the SAN and thus is a multi-domain SSL certificate.

  2. Keystores and trust stores: 
    The certificates are converted to Java Keystore (JKS) format. All the certificates and stores created are kept at same location in each of the servers. These stores are referred in configurations where SSL needs to be enabled by using custom store location.

  1. If we want to ensure zero downtime during certificate renewal, – the file should get processed from one data centre, while the certificate renewal occurs in the other data centre. Then each data centre should have a different certificate and during renewal of certificates in first datacenter (DC1), loadbalancers should be configured to route traffic to the other datacenter (DC2). Secondly, leaf certificates should be removed from all trust stores because root and intermediate certificate(s) are sufficient to validate current as well as future key certificates. The underlying assumption to implement zero downtime is that all key certificates are signed by the same entity (in-house or third party) and thus root and intermediate remain same in foreseeable future. Therefore, to apply new certificates as part of certificate renewal activity, the implementer only needs to replace the current keystore with the new keystore (having the new certificate), restart the applications and perform validation by doing basic smoke testing to ensure file transfer and replication.

  2. Before doing setups in any specific section, please shutdown watchdog (one of the pre-requisites of GM). If watchdog is not shutdown then it will undo the changes in zookeeper configuration files to maintain consistency across nodes. Once all changes are done across nodes then watchdog can be brought up.

Now we will see how to setup secured connections from ZK to other components – other ZK nodes, GM and watchdog.


Zookeeper-to-zookeeper encryption

https://www.ibm.com/support/knowledgecenter/SS3JSW_6.0.3/security/security/globalmailbox/gdha_node_to_node_encryption_zookeeper.html

Node-to-node encryption secures the connections between ZooKeeper servers in the cluster with SSL.

1. In addition to the Quorum TLS that needs to be added to <GM prerequisite install directory>/zookeeper/conf/zoo.cfg file in all servers as follows:



We will also need to add :

ssl.quorum.hostnameVerification=false

 

This disables verification of hostname as common-name in the certificate.

 

2. Restart ZooKeeper

 

3. Start ZooKeeper service on all servers and navigate to<zookeeper install directory>/logs/zookeeper.log to verify that no errors are there in the logs. Please note that since Apache zookeeper is a third-party product, so no specific customization done to verify zookeeper SSL logging. However, on each zookeeper.log one can search for IP addresses of other zookeeper servers in cluster to ensure connectivity is happening without errors. For example:

  ‘grep -i 1.2.3.4 zookeeper.log’

 

B2Bi/GM-to-Zookeeper encryption and authentication

B2Bi/GM acts as a client which tries to establish a secured connection with zookeeper server.

a. Configure ZooKeeper server so that it accepts only secure connections. It can be enabled by editing zoo.cfg file in all servers with the following
    values:

i. Enable the secure client port and comment the client port.

             #clientPort=2181

             secureClientPort=2181

 

ii. Add the below SSL properties as mentioned:

ssl.hostnameVerification=false

The addition of last property disables verification of hostname as common-name in the certificate.


 

b. Configure Global Mailbox to connect to ZooKeeper securely:

 i. encrypted password for a truststore file by going to the following location on B2Bi and GM node:

<SFG installation path>/MailboxUtilities/admin/bin directory.

Run the following command ./sslEncryptionPasswordUtility.sh password

It generates an encrypted password in the format {xor}your_encrypted_password. For example : {xor}E15MWwABGew==

Copy your encrypted password, inclusive of the {xor} characters, and paste it where required onto your property files. Here ‘xor’ denotes the XOR cipher in cryptography

 

ii. Enable SSL on GM client by putting the below in the <config_root>/datacenter.properties file, so the properties apply to all nodes in the datacenter.

com.ibm.mailbox.zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty

com.ibm.mailbox.zookeeper.client.secure=true

com.ibm.mailbox.zookeeper.ssl.trustStore.location<path to truststore>

com.ibm.mailbox.zookeeper.ssl.trustStore.password=<encrypted truststore password>

 

iii. Configure which key Global Mailbox should present when authenticating to ZooKeeper on every Sterling B2B Integrator and IBM Sterling Global
    Mailbox node by adding the following SSL properties:

 

com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

com.ibm.mailbox.zookeeper.ssl.keyStore.location=<path to keystore>

com.ibm.mailbox.zookeeper.ssl.keyStore.password=<encrypted keystore password>

 

into the following files:

 

<B2BI install location>/wlp/usr/servers/mailboxui/mailbox.properties,

<B2BI install location>/MailboxUtilities/admin/conf/mailbox.properties

<B2BI install location>/properties/global_mailbox.properties – For this , please update customer_overrides with prefix :

 

global_mailbox.com.ibm.mailbox.zookeeper.ssl.keyStore.location=<path to keystore>

global_mailbox.com.ibm.mailbox.zookeeper.ssl.keyStore.password=<encrypted keystore password>

global_mailbox.com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

 

4. Restart ZooKeeper

5. Restart the Global Mailbox servers
 

6. Restart the Global Mailbox Client Adapter on every B2Bi node.
 

7. Restart the Sterling B2B Integrator service by running the hardstop.sh and runAll.sh command.

 

8. Verification of SSL connection between GM node and ZK node

As mentioned before, we cannot see SSL specific IBM logging in zookeeper node as it is a third-party product. But we can see it on the client side in <B2BI install directory>/wlp/usr/servers/mailboxui/logs/

 

If ZK connection is not secured then the text will be “Zookeeper Secured Connection is false” else it will be “Zookeeper Secured Connection is true”, “Zookeeper Secured Connection is true. Preparing SSL connection properties.” Some zookeeper configuration properties getting logged can also be seenzookeeper.client.secure, zookeeper.ssl.trustStore.location,

 

trace.log


messages.log



Enabling Watchdog -ZooKeeper encryption and authentication for ZooKeeper

Watchdog can be configured as mentioned in https://www.ibm.com/support/knowledgecenter/SS3JSW_6.0.3/security/security/globalmailbox/gdha_client(watchdog)_to_node(zookeeper)_encryption_zookeeper.html but with one more addition. GMGM

While configuring encryption and authentication by adding com.ibm.mailbox.zookeeper.ssl.* properties listed below in the <GM Pre-requisite install location>/zookeeper/watchdog/conf/watchdog.properties

 

also add the property :

com.ibm.mailbox.zookeeper.ssl.hostnameVerification=false

 

c. Restart the watchdog service 


By keeping the above mentioned information at hand and implementing the configuration, we achieve secured connections between Apache Zookeeper and other components in an enterprise  level IBM Sterling Global Mailbox setup using multi-domain certificates and ensuring minimal downtime.








#Featured-area-1
#Featured-area-1-home
0 comments
322 views

Permalink