In this article, I will describe the new KafkaProducer and KafkaConsumer nodes that have been provided in IBM Integration Bus 10.0.0.7. I will show how you can publish messages to a topic on IBM Message Hub and consume messages from that topic.

Overview of Apache Kafka

Apache Kafka is a distributed streaming platform which allows applications to publish and subscribe to streams of records. Kafka is architected as a cluster of one or more servers. A stream of records is called a ‘topic’. A ‘Producer’ can publish messages to a topic. A ‘Consumer’ can subscribe to a topic and consume the messages that were published.

The above diagram shows that when messages are published on a Kafka topic, the messages are added to the tail of the log. As each message is published on the topic, it is identified by the ‘offset’ of the message in the log. All consumers retrieve messages from the same log, and as messages are consumed they are not destroyed but remain in the log for the pre-defined retention period. The offset is used by consumers to identify their position when receiving messages. The retention period can be defined by time, or by log size.

(see attached PDF for full article)