WebSphere Application Server & Liberty

This article aims to demonstrate the following:

• How to configure WebSphere Liberty to use CCDT (client channel definition table) to connect to the MQ queue manager.
• How to use CCDT to automatically reconnect to another queue manager if the connection fails, without writing any client code.
• Understand the behavioural difference when used in JMS Connection Factory (sending messages to a queue via JMS) and Activation Specification (receiving messages from a queue via MDB)

This article is not intended to provide details of all the concepts used here. However, it will reference elsewhere so that you can do some prior reading to get a deeper understanding. Before we start, let's have a short brief on some of the concepts and definitions.

• CCDT or client channel definition table is a file (MQ) that contains information needed by a client application to connect to a queue manager or in a queue manager group. It contains information such as connection names, channel definitions, authentication information and TLS cipher specifications. In MQ v9.1, we can use a JSON-format CCDT. See here for more details.
• A queue manager group is used to implement a failover configuration - which has one primary queue manager and multiple backup queue managers. You can use CCDT to configure queue manager groups so that failover using the MQ automatic client reconnection (without any code changes in the application). Learn more about the queue manager group here.
• However, this can only be used in the outbound scenario (JMS Connection Factory) and not in the inbound scenario (Activation Specification). See here for more explanation.

Some of the pre-requisites:

• Download WebSphere Liberty (wlp-kernel-YY.n.n.n.zip) from here. I am using wlp-kernel-22.0.0.9.zip. Note the ccdtURL is not supported on Open Liberty.
• Download IBM MQ Resource Adapter from Fix Central. I am using 9.3.0.1-IBM-MQ-Java-InstallRA.jar.
• Install Java
• Install Docker
• Install curl
• Install Visual Studio Code  (if you want to do code changes)

Architecture

The diagram below shows the setup in the article.

• There is one JMS Connection Factory configured with a CCDT that points to two client channel definitions - with the queue manager group "QMGRP". The JMS Connection Factory passes in the "*QMGRP". When prefixed with an asterisk then any queue manager is a possible connection target that is associated with the queue manager group "QMGRP". 
  

  {
      "channel": [
        {
          "name": "DEV.APP.SVRCONN",
          "clientConnection": {
            "connection": [
              {
                "host": "localhost",
                "port": 1414
              }
            ],
            "queueManager": "QMGRP"
          },
          "type": "clientConnection"
        },
        {
          "name": "DEV.APP.SVRCONN",
          "clientConnection": {
            "connection": [
              {
                "host": "localhost",
                "port": 1415 
              }
            ],
            "queueManager": "QMGRP"
          },
          "type": "clientConnection"
        }
      ]
  }

• There are two message-driven beans (MDB) configured with corresponding CCDT via the Activation Specification. We configured only one client channel definition per MDB because Activation Specification does not support queue manager group setup. Therefore, we have two MDBs, one receiving messages from the queue manager QM1 and QM2 respectively.
  

  //////////////////////////////////////////////////////////
  // ccdtQM1.json
  //////////////////////////////////////////////////////////
  {
      "channel": [
        {
          "name": "DEV.APP.SVRCONN",
          "clientConnection": {
            "connection": [
              {
                "host": "localhost",
                "port": 1414
              }
            ],
            "queueManager": "QM1"
          },
          "type": "clientConnection"
        }
      ]
  }
  
  //////////////////////////////////////////////////////////
  // ccdtQM2.json
  //////////////////////////////////////////////////////////
  {
      "channel": [
        {
          "name": "DEV.APP.SVRCONN",
          "clientConnection": {
            "connection": [
              {
                "host": "localhost",
                "port": 1414
              }
            ],
            "queueManager": "QM1"
          },
          "type": "clientConnection"
        }
      ]
  }​

• We will run the queue managers in a Docker container, and the WebSphere Liberty server in the host machine.

 Get the artefacts from the Git repository

1. Clone the git repository
   

   git clone https://github.com/khongks/ws-liberty-mq-ccdt.git

Install WebSphere Liberty

1. Un-zip the downloaded zip file into a folder called wlp.
2. Go to folder wlp.
3. Create a server. This creates the folder usr/servers/testServer (under wlp folder) that contains all the runtime files needed to run the server.
   

   bin/server create testServer​

4. Copy the server.xml into the folder usr/servers/testServer, replacing the default server.xml
5. Install all features for testServer.
   

   bin/featureUtility installServerFeatures testServer​

Configure MQ files

1. Unpack the JAR file
   

   jar xvf 9.3.0.1-IBM-MQ-Java-InstallRA.jar​​

2. Copy the file wmq/wmq.jmsra.rar into the folder wlp/usr/shared/resources.
3. Copy the CCDT files (ccdt.json, ccdtQM1.json, ccdtQM2.json) into wlp/usr/shared/resources.

Understand WebSphere Liberty configuration

1. All the configurations in WebSphere Liberty are defined in server.xml.
2. Add the required features.
   

       <featureManager>
           <feature>javaee-7.0</feature>
           <feature>mdb-3.2</feature>
           <feature>wmqJmsClient-2.0</feature>
           <feature>jndi-1.0</feature>
           <feature>localConnector-1.0</feature>
       </featureManager>​

3. Configure MQ Resource Adapter. For more information on the Liberty folders and properties mapping, see here.
   

       <variable name="wmqJmsClient.rar.location" value="${shared.resource.dir}/wmq.jmsra.rar"/>​

4. Configure JMS Connection Factory. It is important to note that the ccdtURL field is only supported in WebSphere Liberty, not Open Liberty. I am specifying "*QMGRP" in the queueManager field to make use of the queue manager group feature. T
   

       <jmsConnectionFactory connectionManagerRef="ConMgr6" jndiName="jms/wmqCF">
           <properties.wmqJms ccdtURL="file://${shared.resource.dir}/ccdt.json" password="{xor}Lz4sLChvLTs=" queueManager="*QMGRP" userName="app"/>
       </jmsConnectionFactory>​

5. Configure Activation Specifications
   

       <jmsActivationSpec authDataRef="mqAuth" id="mdbtest/SampleListenerMDBQM1">
           <properties.wmqJms 
           ccdtURL="file://${shared.resource.dir}/ccdtQM1.json"         
           queueManager="QM1" 
           destinationRef="jms/queue1"/>
       </jmsActivationSpec>
       
       <jmsActivationSpec authDataRef="mqAuth" id="mdbtest/SampleListenerMDBQM2">
           <properties.wmqJms 
           ccdtURL="file://${shared.resource.dir}/ccdtQM2.json" 
           queueManager="QM2"
           destinationRef="jms/queue1"/>
       </jmsActivationSpec>​

6. Configure JMS Queue
   

       <jmsQueue id="jms/queue1" jndiName="jms/queue1">
           <properties.wmqJms baseQueueName="APP.Q"/>
       </jmsQueue>​

7. Install the features
   

   bin/featureUtility installServerFeatures testServer

8. Configure the reconnection behaviour of the MQ Resource Adapter. When the connection to a queue manager is down, Activation Specification will try to reconnect 360 times every 5 seconds before it stops. You will have to restart the server to reestablish a connection to the queue manager again. This means you have 30 mins to recover the queue manager. You can tune these values according to your requirements.
   

       <wmqJmsClient reconnectionRetryCount="3600" reconnectionRetryInterval="5000" startupRetryCount="3600" startupRetryInterval="1000ms">
       </wmqJmsClient>​

Prepare and run two queue managers

1. In the ibmmq folder, there is a Dockerfile and queue manager configuration file 20-config.mqsc. The MQSC file defines two key objects that will be used by the application - the channel DEV.APP.SVRCONN and the local queue APP.Q. Both queue managers have the same configuration.
   

   DEFINE CHANNEL(DEV.APP.SVRCONN) CHLTYPE(CLNTCONN) CONNAME('localhost(1414)') QMNAME('QMGRP') USERID('app') PASSWORD('passw0rd') REPLACE
   DEFINE QLOCAL(APP.Q) REPLACE
   SET AUTHREC OBJTYPE(QUEUE) PROFILE(APP.Q) PRINCIPAL('app') AUTHRMV(ALL) AUTHADD(PUT,GET,BROWSE,INQ)​

2. Build the image.
   

   docker build -t myqueuemanager:1.0 .​

3. Run queue manager QM1 exposing ports 1414 for MQ traffic and 9443 for MQ console.
   

   docker run --env LICENSE=accept --env MQ_QMGR_NAME=QM1 --publish 1414:1414 --publish 9443:9443 --detach --env MQ_APP_PASSWORD=passw0rd --name qm1 myqueuemanager:1.0​

   
   

4. Run queue manager QM2 exposing ports 1415 for MQ traffic and 9444 for MQ console.
   

   docker run --env LICENSE=accept --env MQ_QMGR_NAME=QM2 --publish 1415:1414 --publish 9444:9443 --detach --env MQ_APP_PASSWORD=passw0rd --name qm2 myqueuemanager:1.0

   
   

To compile the application

1. Compile the source code and generate a war file. Go to the mdbtest folder and run this command.
   

   ./mvnw install​

   
   

2. The war file is generated in the target folder. Copy the mdbtest.war file into wlp/usr/servers/testServer/dropins.

Start application server

1. Start the server
   

   bin/server start testServer

   
   
   
2. Check the wlp/usr/servers/testServer/logs/message.log. You will note that both ActivationSpec(s) are connected.
   

   [1/10/22, 1:14:34:421 AEST] 0000002b WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM2' connected or reconnected (to host 'localhost/127.0.0.1':'1415' - 'QM2') successfully.
   [1/10/22, 1:14:34:422 AEST] 0000002b com.ibm.ws.jca.service.EndpointActivationService             I J2CA8801I: The message endpoint for activation specification mdbtest/SampleListenerMDBQM2 and message driven bean application mdbtest#mdbtest.war#SampleListenerMDBQM2 is activated.
   [1/10/22, 1:14:34:519 AEST] 0000002b WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connected or reconnected (to host 'localhost/127.0.0.1':'1414' - 'QM1') successfully.
   [1/10/22, 1:14:34:520 AEST] 0000002b com.ibm.ws.jca.service.EndpointActivationService             I J2CA8801I: The message endpoint for activation specification mdbtest/SampleListenerMDBQM1 and message driven bean application mdbtest#mdbtest.war#SampleListenerMDBQM1 is activated.
   [1/10/22, 1:14:34:554 AEST] 0000002b com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0012I: The server installed the following features: [appClientSupport-1.0, appSecurity-2.0, batch-1.0, beanValidation-1.1, cdi-1.2, concurrent-1.0, distributedMap-1.0, ejb-3.2, ejbHome-3.2, ejbLite-3.2, ejbPersistentTimer-3.2, ejbRemote-3.2, el-3.0, j2eeManagement-1.1, jacc-1.5, jaspic-1.1, javaMail-1.5, javaee-7.0, jaxb-2.2, jaxrs-2.0, jaxrsClient-2.0, jaxws-2.2, jca-1.7, jcaInboundSecurity-1.0, jdbc-4.1, jms-2.0, jndi-1.0, jpa-2.1, jpaContainer-2.1, jsf-2.2, json-1.0, jsonp-1.0, jsp-2.3, localConnector-1.0, managedBeans-1.0, mdb-3.2, servlet-3.1, ssl-1.0, wasJmsClient-2.0, wasJmsSecurity-1.0, wasJmsServer-1.0, webProfile-7.0, websocket-1.1, wmqJmsClient-2.0].
   [1/10/22, 1:14:34:556 AEST] 0000002b com.ibm.ws.kernel.feature.internal.FeatureManager            I CWWKF0008I: Feature update completed in 5.919 seconds.
   [1/10/22, 1:14:34:556 AEST] 0000002b com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0011I: The testServer server is ready to run a smarter planet. The testServer server started in 6.926 seconds.
   ​

Time to test

1. Send a message.
   

   curl -kv -X POST -d 'msg=test message' http://localhost:9080/mdbtest/api/enqueue​

2. Check the logs for messages received from QM1.
   

   [1/10/22, 1:20:05:387 AEST] 0000003b .ibm.ws.jaxrs.2.0.common:1.0.68.cl220920220815-1900(id=282)] I Setting the server's publish address to be /api/
   [1/10/22, 1:20:05:590 AEST] 0000003b com.ibm.ws.webcontainer.servlet                              I SRVE0242I: [mdbtest] [/mdbtest] [com.demo.rest.RestApplication]: Initialization successful.
   [1/10/22, 1:20:05:795 AEST] 0000003b com.ibm.ws.jca.cm.ConnectorService                           I J2CA8050I: An authentication alias should be used instead of defining a user name and password on jms/wmqCF.
   [1/10/22, 1:20:06:294 AEST] 0000003b SystemOut                                                    O Message enqueued.
   [1/10/22, 1:20:06:421 AEST] 00000062 SystemOut                                                    O MDB received from QM1: test message​

3. Stop the queue manager QM1.
   

   docker stop qm1​

4. Check the logs for errors and  reconnection attempts.
   

        Caused by [4] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host 'localhost/127.0.0.1:1414' rejected. [1=java.net.ConnectException[Connection refused (Connection refused)],3=localhost/127.0.0.1:1414,4=TCP,5=Socket.connect]
                            Class : class com.ibm.mq.jmqi.JmqiException
                            Stack : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:932)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1469)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:1011)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:688)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:282)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:181)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13375)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(RemoteFAP.java:13175)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1449)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1390)
                                  : com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
                                  : com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
                                  : com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:391)
                                  : com.ibm.msg.client.wmq.internal.WMQXAConnection.<init>(WMQXAConnection.java:67)
                                  : com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:187)
                                  : com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:8023)
                                  : com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createProviderXAConnection(WMQXAConnectionFactory.java:98)
                                  : com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createXAConnectionInternal(JmsConnectionFactoryImpl.java:385)
                                  : com.ibm.mq.jms.MQXAConnectionFactory.createXAConnection(MQXAConnectionFactory.java:104)
                                  : com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:296)
                                  : com.ibm.mq.connector.inbound.MessageEndpointDeployment.acquireConnection(MessageEndpointDeployment.java:399)
                                  : com.ibm.mq.connector.inbound.ConnectionHelperThread.run(ConnectionHelperThread.java:136)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
                                  : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
                                  : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1240)
        Caused by [5] --> Message : java.net.ConnectException: Connection refused (Connection refused)
                            Class : class java.net.ConnectException
                            Stack : java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:-2)
                                  : java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
                                  : java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
                                  : java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
                                  : java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
                                  : java.net.Socket.connect(Socket.java:609)
                                  : java.net.Socket.connect(Socket.java:558)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1103)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1095)
                                  : java.security.AccessController.doPrivileged(AccessController.java:-2)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connectSocket(RemoteTCPConnection.java:1095)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:838)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1469)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:1011)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:688)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:282)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:181)
                                  : com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13375)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(RemoteFAP.java:13175)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1449)
                                  : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1390)
                                  : com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
                                  : com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
                                  : com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:391)
                                  : com.ibm.msg.client.wmq.internal.WMQXAConnection.<init>(WMQXAConnection.java:67)
                                  : com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:187)
                                  : com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:8023)
                                  : com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createProviderXAConnection(WMQXAConnectionFactory.java:98)
                                  : com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createXAConnectionInternal(JmsConnectionFactoryImpl.java:385)
                                  : com.ibm.mq.jms.MQXAConnectionFactory.createXAConnection(MQXAConnectionFactory.java:104)
                                  : com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:296)
                                  : com.ibm.mq.connector.inbound.MessageEndpointDeployment.acquireConnection(MessageEndpointDeployment.java:399)
                                  : com.ibm.mq.connector.inbound.ConnectionHelperThread.run(ConnectionHelperThread.java:136)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
                                  : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
                                  : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
                                  : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1240)
   '.
   [1/10/22, 1:28:30:019 AEST] 000000ae WMQ.commonservices.trace                                     I Activation specification reconnection attempt '1' of '3600'.  Waiting '5000' milliseconds before attempting to reconnect the endpoint 'mdbtest#mdbtest.war#SampleListenerMDBQM1'.
   [1/10/22, 1:28:35:024 AEST] 000000ae WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connect failed due to error 'JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').' (queue manager: 'QM1', host : 'localhost', port: '1414', channel: 'SYSTEM.DEF.SVRCONN')
   [1/10/22, 1:28:35:024 AEST] 000000ae WMQ.commonservices.trace                                     I Activation specification reconnection attempt '2' of '3600'.  Waiting '5000' milliseconds before attempting to reconnect the endpoint 'mdbtest#mdbtest.war#SampleListenerMDBQM1'.
   [1/10/22, 1:28:40:030 AEST] 000000ae WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connect failed due to error 'JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').' (queue manager: 'QM1', host : 'localhost', port: '1414', channel: 'SYSTEM.DEF.SVRCONN')
   [1/10/22, 1:28:40:031 AEST] 000000ae WMQ.commonservices.trace                                     I Activation specification reconnection attempt '3' of '3600'.  Waiting '5000' milliseconds before attempting to reconnect the endpoint 'mdbtest#mdbtest.war#SampleListenerMDBQM1'.
   [1/10/22, 1:28:45:034 AEST] 000000ae WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connect failed due to error 'JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').' (queue manager: 'QM1', host : 'localhost', port: '1414', channel: 'SYSTEM.DEF.SVRCONN')
   [1/10/22, 1:28:45:035 AEST] 000000ae WMQ.commonservices.trace                                     I Activation specification reconnection attempt '4' of '3600'.  Waiting '5000' milliseconds before attempting to reconnect the endpoint 'mdbtest#mdbtest.war#SampleListenerMDBQM1'.
   [1/10/22, 1:28:50:040 AEST] 000000ae WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connect failed due to error 'JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').' (queue manager: 'QM1', host : 'localhost', port: '1414', channel: 'SYSTEM.DEF.SVRCONN')
   [1/10/22, 1:28:50:051 AEST] 000000ae WMQ.commonservices.trace                                     I Activation specification reconnection attempt '5' of '3600'.  Waiting '5000' milliseconds before attempting to reconnect the endpoint 'mdbtest#mdbtest.war#SampleListenerMDBQM1'.​

5. Send another message.
6. Check the logs for messages received from QM2. Explanation: because QM1 is down, the application configured in the CCDT as queue manager group, uses the automatic client reconnection to send the message to QM2.
   

   [1/10/22, 1:29:34:524 AEST] 0000009a SystemOut                                                    O Message enqueued.
   [1/10/22, 1:29:34:608 AEST] 000000b8 SystemOut                                                    O MDB received from QM2: test message​

7. Start the queue manager QM1.
   

   docker start qm1​

8. Check the logs to find that ActivationSpec is reconnected.
   

   [1/10/22, 1:30:40:377 AEST] 000000ae WMQ.commonservices.trace                                     I ActivationSpec 'mdbtest#mdbtest.war#SampleListenerMDBQM1' connected or reconnected (to host 'localhost/127.0.0.1':'1414' - 'QM1') successfully.​

9. Send another message.
10. Check the logs for messages received from QM2. Explanation: because while QM1 is up again, the application continues to connect to queue manager QM2.
    

    [1/10/22, 1:31:47:826 AEST] 000000b0 SystemOut                                                    O Message enqueued.
    [1/10/22, 1:31:47:838 AEST] 0000008e SystemOut                                                    O MDB received from QM2: test message​

11. Stop the queue manager QM2.
    

    docker stop qm2​

12. Send another message.
13. Check the logs for messages received from QM1. Explanation: because QM2 is down, the application now reconnects to QM1.
    

    [1/10/22, 1:32:39:806 AEST] 00000030 SystemOut                                                    O Message enqueued.
    [1/10/22, 1:32:39:890 AEST] 00000098 SystemOut                                                    O MDB received from QM1: test message​

14. Start the queue manager QM2.
    

    docker start qm2​

Thank you for reading.