Bridging IBM MQ and Apache Kafka

Sampoorna Varshinii M.K.S

Technical Support Specialist – MQ & Event Automation 

Moving legacy or transactional messaging workloads into an event-driven ecosystem opens massive possibilities for real-time analytics and microservices architecture. This article demonstrates how to use Kafka Connect to bridge IBM MQ and event streams, creating a reliable data pipeline that captures MQ messages.

Introduction 

This technical note describes configuring an end-to-end data pipeline where IBM MQ on OpenShift Container Platform (OCP) serves as the source system and IBM Event Streams acts as the streaming platform. Kafka Connect source connectors are used to reliably stream data from MQ to Event Streams.

Key Components:  IBM MQ Queue Manager, Kafka Connect with MQ Source plugin & IBM Event Streams (Kafka). In the examples below, I have used version: OpenShift 4.19.26, deployed instances of IBM Event Streams 12.2.2 and IBM MQ 9.4.4.0-r3, plus the oc CLI and keytool 11.0.30 utilities.

Architectural overview

Log in to the OpenShift cluster, switch to the target namespace, and run oc get eventstreams to verify the Event Streams instance before proceeding.

Authentication:

1. Create MQ Authentication Secret :
2. oc create secret generic <secret-name> --from-literal=username=<mq-username> --fromliteral=password=<mq-password> -n <namespace>

            For eg: oc create secret generic mq-source-credentials --from-literal=username=app -from-literal=password=passw0rd -n test-ns

Alternative method: Create a secret to store MQ authentication credentials using the YAML

2. Create Kafka User for Connect Authentication: Substitute the name of the user here test-connect and namespace appropriately. Note the type of authentication and ensure proper ACL has been configured for the user. 

3.   Apply the configuration: This automatically creates a secret `test-connect` with the password: oc apply -f kafka-user-test-connect.yaml

MQ Source Connector Setup

1.    Create the Kafka Connect cluster with MQ Source plugin. The major parts of the configuration that involve changes are highlighted here.

Refer to the connect catalog to get the proper type and url.

https://ibm.github.io/event-automation/connectors/kc-source-ibm-mq/installation

https://ibm.github.io/event-automation/connectors/converter-apicurio-avro/installation

2. Apply the configuration: oc apply -f mq-kafka-connect.yaml

3.Verify the Kafka Connect instance: (The expected status is to be READY)

oc get kafkaconnect <connect-cluster-name> -n <namespace>

Eg: oc get kafkaconnect sach-mq-connect-connect -n test-ns

4. Create MQ Source Connector

Before configuring the connector, ensure that you have a MQ queue manager running. Create the MQ objects (queuemanager, local queue, svrconn channel) that will be used to communicate to the kafka (topic) either manually over the terminal using the runmqsc commands or using the config map.

4a. ConfigMap YAML file: The data part is to be focused on and defined as per the requirement

The setup includes a local queue (DEV.QUEUE.1) and a server connection channel (KAFKA.CONN.SVRCONN), along with the required authentication records configured for the user defined in the mq-source-credentials file. 

4b. Once you have the MQ objects defined, next step is to create MQ Source Connector YAML file with appropriate configuration. For more details on the corresponding parameters to be substituted please refer document.

https://ibm.github.io/event-automation/es/es_12.2/connecting/mq/source/

And as per our MQ definitions the connector YAML file should finally look like this:   

This configuration uses the com.ibm.eventstreams.connect.mqsource.MQSourceConnector class to stream raw byte data (via ByteArrayConverter) from the DEV.QUEUE.1 queue on queue manager sachmq into the Kafka topic mq-sourcetopic. It connects via channel KAFKA.CONN.SVRCONN at endpoint sachmq-ibm-mq(1414), securely authenticating using a mounted credentials file. Built using DefaultRecordBuilder, the connector is currently running with one active task, with a customizable batch size of up to 250 messages.

5. If you are creating through oc CLI then proceed with the below command to apply the changes in the configuration:

oc apply -f mq-source-connector.yaml

6.  Verify the connector : oc get kafkaconnector <connector-name> -n <namespace> 

Here: oc get kafkaconnector mq-source-connector -n test-ns 

7. If you are creating through the OCP UI then click on create, if at all you make changes after you create the basic YAML, procced to save and reload the to apply the changes.

For more details refer: https://ibm.github.io/event-automation/es/connecting/mq/source/

TESTING 

Now that we have configured both MQ Source let us test the message flow. It can either be done using the oc CLI or through the OCP UI. 

         1.  Send message to MQ queue:

1a. Using CLI: oc exec <mq-pod-name> -n <namespace> -- bash -c "echo '<your-message>' | /opt/mqm/samp/bin/amqsput <queue-name> <queue-manager-name>"

1b. Using UI: Login to OCP cluster and execute the execute the commands in the MQ pod.

         2.     Verify Messages in Event Streams:

2a. Open the ES UI using the link available on the right-hand side of your ES instance.

2b. Log in with the Kafka user (test-connect). Navigate to Resources, select the resource, and copy the generated password.

2c. In the ES UI, go to Topics → mqsource-topic → Messages, then refresh the page to view the messages. The latest message sent in the previous step should be visible.

NOTE: 

Performance and tuning parameters should be adjusted based on the deployment environment, workload, message volume, system resources, network latency, and downstream system capacity. Settings such as batch size, polling interval, task count, retries, and Kafka Connect resource allocation should be fine-tuned through testing to ensure optimal throughput, latency, and resource utilisation.