Introduction

Kafka follows a publish–subscribe model, where producers send messages to topics and consumers read from them. A topic represents a stream of events and is divided into partitions for scalability. A Kafka cluster consists of multiple brokers working together to provide scalability and fault tolerance.

• Topic → Partition 0, Partition 1, Partition 2
• Each partition contains messages ordered by offset
• Messages are stored as key-value pairs, where:
  • Key = metadata
  • Value = actual message data

IBM MQ, on the other hand, provides a highly reliable, enterprise-grade messaging backbone built on a queue-based model, enabling asynchronous communication between applications. It is designed for mission-critical workloads and ensures:

• Guaranteed and persistent message delivery
• Transactional integrity across systems
• Loose coupling between applications and services

To integrate this reliability with Kafka’s streaming capabilities, Kafka Connect provides a standardised framework for moving data between Kafka and enterprise systems like MQ.

With IBM MQ Advanced, Kafka Connect includes:

• Source connector → MQ → Kafka
• Sink connector → Kafka → MQ

Architecture

In this setup:

• Kafka runs locally on your machine (Windows, macOS, or Linux)
• Kafka Connect runs in standalone mode (locally and on z/OS USS)
• The MQ Source connector reads messages from MQ queues and publishes them to Kafka topics
• The MQ Sink connector consumes messages from Kafka topics and writes them to MQ queues

Message flow:

• MQ Queue → Kafka Topic
• Kafka Topic → MQ Queue

Setting up Kafka

Download and extract Kafka from https://kafka.apache.org/community/downloads/

Open the terminal and extract the Kafka package and start it

tar -xzf kafka_2.13-4.2.0.tgz
bin/kafka-server-start.sh config/server.properties 

If Kafka fails with: No readable meta.properties files found, initialise storage (KRaft mode) and restart Kafka

bin/kafka-storage.sh random-uuid
bin/kafka-storage.sh format -t <UUID> -c config/server.properties --standalone
bin/kafka-server-start.sh config/server.properties
[2026-03-01 13:11:05,917] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer) 

Setting up the MQ Source Connector

Clone and build:

git clone https://github.com/ibm-messaging/kafka-connect-mq-source.git
mvn clean package

Sample Configuration

name=mq-source
connector.class=com.ibm.eventstreams.connect.mqsource.MQSourceConnector
tasks.max=1
topic=KAFKA.TOPIC
mq.queue.manager=QMGR
mq.connection.mode=client
mq.connection.name.list=hostname(port)
mq.channel.name=KAFKA.SVRCONN
mq.queue=SOURCE.QUEUE
mq.record.builder=com.ibm.eventstreams.connect.mqsource.builders.DefaultRecordBuilder
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

Running the Source Connector:

./bin/connect-standalone.sh \
  config/connect-standalone.properties \
  kafka-connect-mq-source/config/mq-source.properties

Consuming Messages

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic KAFKA.TOPIC \
  --from-beginning

If the topic does not exist:

bin/kafka-topics.sh \
  --create \
  --topic KAFKA.TOPIC \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

Handling Message Format:

If messages appear like: [B@28375826, this is due to byte-array handling. You can fix this by using:

key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

Setting up the MQ Sink Connector

git clone https://github.com/ibm-messaging/kafka-connect-mq-sink.git

Sample Configuration

topics=KAFKA.TOPIC
mq.queue.manager=QMGR
mq.connection.mode=client
mq.connection.name.list=hostname(port)
mq.channel.name=KAFKA.SVRCONN
mq.queue=SINK.QUEUE
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter

Running the Sink Connector

./bin/connect-standalone.sh \
  config/connect-standalone-sink.properties \
  kafka-connect-mq-sink/config/mq-sink.properties

If a port conflict occurs, use a different available port to allow the connection to MQ to be established and the sink task to initialize.

rest.port=8084

End-to-End Testing: 

Kafka → MQ

bin/kafka-console-producer.sh \
  --topic KAFKA.TOPIC \
  --bootstrap-server localhost:9092

or

echo "Test Message from Kafka" | bin/kafka-console-producer.sh --topic KAFKA.TOPIC --bootstrap-server localhost:9092

Browse the message in MQ Explorer

MQ → Kafka

Put a message into the MQ queue using MQ Explorer (or any MQ client tool) → appears in Kafka consumer.

Running Kafka Connect on z/OS

From IBM MQ 9.4.3 onwards, the Connector Pack provides a supported Kafka Connect runtime on z/OS UNIX System Services.

This runtime is intended specifically for running MQ connectors.

Navigate in z/OS: 

ISPF → Option 6 → OMVS

Example directory and files:

/mqm/V9R4M5/kafka-connect:>ls -ltr
drwxrwxr-x   2 MQZBLD   VICYCT      8192 Jan 15 19:35 xml-converter
drwxrwxr-x   2 MQZBLD   VICYCT      8192 Jan 15 19:35 sink
drwxrwxr-x   2 MQZBLD   VICYCT      8192 Apr  7 15:24 source
drwxrwxr-x   7 MQZBLD   VICYCT      8192 Apr  7 15:49 kafka-connect

Connector files allow Kafka Connect to run directly within z/OS USS.

Set Environment Variables:

This is mandatory on z/OS:

export STEPLIB=hlq.SCSQAUTH:hlq.SCSQANLE:$STEPLIB
export LIBPATH=/usr/lpp/java/J8.0/lib:$LIBPATH
export PATH=$PATH:/mqm/V9R4M5/kafka-connect/kafka-connect/bin

Replace hlq with your MQ high-level qualifier if needed.

Update connect-standalone.properties

vi config/connect-standalone.properties
bootstrap.servers=your.kafka.server:9092
plugin.path=/mqm/V9R4M5/kafka-connect/source,/mqm/V9R4M5/kafka-connect/sinkyour.kafka.server

Replace your.kafka.server with your local server IP/hostname where your Kafka broker is running, and your z/OS system must be able to connect to it.

#Source connector
./bin/connect-standalone.sh  config/connect-standalone.properties  ../source/mq-source.properties

#Sink connector
./bin/connect-standalone.sh  config/connect-standalone.properties  ../sink/mq-sink.properties

Summary

This setup demonstrates how Kafka Connect bridges modern event streaming with enterprise messaging systems.

• MQ provides reliable message delivery
• Kafka enables scalable event streaming
• Kafka Connect enables seamless integration

The queue manager can be hosted on any platform, including distributed systems or z/OS. From a z/OS MQ perspective, Kafka Connect can also run natively in USS using the Connector Pack, providing greater deployment flexibility. This combination allows organisations to integrate modern and legacy systems efficiently, without requiring complex custom development