Plenty has been said and written about the comparison between messaging capabilities like IBM MQ and event streaming capabilities such as Apache Kafka, but perhaps there’s not enough out there emphasising how these are really complementary technologies and why.

It turns out that the data flowing across a messaging network such as IBM MQ is exceedingly well suited to providing events for Apache Kafka.

IBM has had messaging and Apache Kafka alongside one another in our portfolio for a long time. Our acquisition of Confluent this year makes understanding how these technologies work together all the more important.

In this article we’re going to explore a number of reasons why it would be beneficial to use IBM MQ to cover the distance and disparities between your application and an Apache Kafka deployment rather than having your application talk directly to Kafka.

You may have an existing MQ landscape, and someone is trying to convince you that you should now be doing everything in Kafka – that you don’t need messaging anymore. It’s critical to recognise that messaging is perfectly suited to what it is already doing, and Apache Kafka is a complementary technology, not a competing one. Indeed, this article articulates how IBM MQ can actually be an accelerator to successful Kafka adoption.

Messaging sees all the best events

Business transactions delegated to asynchronous messaging, tend to be the “big hitters”. This is especially true where IBM MQ is involved as it is typically chosen as a safe pair of hands due to its assured, exactly once, secure delivery across a wide variety of different industries.

The very fact that a message is being sent between systems means it is likely that it represents something important. IBM MQ is typically used to reliably transmit “commands” to request that another system perform an action. These often represent key transitions in the state of core business entities and processes, such as the submission of an order or the processing of a payment.

Apache Kafka’s speciality is gaining insight from events in real time, enabling the business to react more quickly and remain more competitive. What better place to gain that insight than from the core transactions on which your business depends.

Messaging is chosen for the most critical transactions to the business: payments, orders, claims, shipments, bookings, invoices and more. For that same reason they are likely to be some of the most valuable to also surface as events in Apache Kafka.

Message replication is non-disruptive

Creating events from existing IBM MQ interactions does not require any additional coding, nor indeed any changes to the surrounding applications currently producing and consuming messages via the queue.

Queues can be configured to send copies of the messages to a separate “streaming queue”. These are enabled by a simple non-disruptive change to the queue definition on a queue manager.

Application A might for example be sending payments for processing by Application B, and we want to create an additional feed of payment information in order to look for fraudulent payments. However, we don’t want to make any alterations to the two applications performing such a business-critical transaction. With a simple configuration on the queue manager, we can introduce a streaming queue for the fraud check with no risk of interruption of the payments.

The streaming queue is just like any other IBM MQ queue and can be used to transport the messages to anywhere within the enterprise via the MQ messaging network with the assured exactly once delivery you would expect.

To ensure critical transactions are not affected by the attempt to copy to the streaming queue, the default behaviour is for the original message PUT to succeed even if the copy to the streaming queue fails. However, it is also possible to enforce that the copy is made as there may be circumstances where it is essential that the streaming queue is updated. An example might be when creating a security audit trail.

If the end destination for the streaming queue is Apache Kafka then there is an MQ to Kafka connector provided as part of the IBM MQ Advanced license. This connector is built on the Kafka Connect framework used for all Apache Kafka connectors and has been specifically engineered to ensure exactly once delivery onto a Kafka topic. The MQ/Kafka connector can of course be used with any IBM MQ queue, not just ones defined as streaming queues.

It’s also worth being aware that the connector is bi-directional. So, if for example as a result of streaming analytics a particular insight requires action to be taken, Kafka can create events that can be reliably translated into messages.

Many other capabilities such as IBM App Connect, and CICS applications can also be configured to produce events to IBM MQ without changes to the integrations or code. Which events are emitted and what they contain can typically also be configured.

Now since there are many different potential sources of events, let’s look at why, specifically, IBM MQ may be one of the best.

Messages know the context

Messages sent between applications typically contain more contextually rich information than you might get by listening in to point data sources. Perhaps one of the most striking examples of this is when you compare typical content of an IBM MQ message with what you might get if you used change data capture (CDC) to pick up changes directly from a database.

Change data capture (CDC) approach: A common way to capture key business events occurring in your underlying systems is to make use of a change data capture mechanism. This listens to a specific table in a database and advises you of any added, changed or deleted rows. This, at first glance, may look like the simpler approach, but it results in a much less valuable business event. What do these table change events tell you about the overall business transaction that was taking place? The business transaction may be across multiple tables, and CDC has no sight of that.

Message copy approach: Important business events typically result in one application asking another application to perform a unit of work, and messaging is often involved to ensure reliable delivery of that request.  When messages are sent between applications, they typically carry all the pertinent context for the action to be completed on the other side. Message payloads are often a “graph” of information including all of the data from parent and child objects needed to fully understand the business transaction taking place. A message representing the business transaction will at the very least contain foreign keys into other tables and may well contain the actual data from them too.

In a greatly over-simplified example, we could consider an “orders” table. In a normalised data structure, the table almost certainly won’t contain “order items”, nor will it contain any data about the customer placing the order. However, if we were to intercept a “submit order” message being sent between applications, it will often have all that information and more.

Furthermore, messages may contain more context than can be found in any one application’s database, as it may contain information related to multiple applications. This is especially true if you are able to tap into the messages relating to an integration flow that is co-ordinating a multi-application update. The message may be the only place where you will find all the context in one place.

In summary, sourcing events for Kafka from within your MQ messaging network can make it easier to access more complete event data that better represents your key business transactions.

Messaging networks are designed to span distributed landscapes

Apache Kafka is a relatively significant deployment, so often we see quite centralized installations. As a result, the location of your Apache Kafka deployment will be different from the application(s) producing events. Do you want to set up connectivity for each of your applications so they can make direct connections to Apache Kafka? If those connections go down, are you ok with losing the ability to record events for that period? A good alternative is to use an IBM MQ messaging network to do the event transportation, providing a light footprint local delivery point that can then reliably transport the events to Kafka without your applications needing to know how that is done.

Producing events first to a messaging network and letting that get the events to the Apache Kafka deployment location provides a number of benefits:-

• The applications don’t need to know where Kafka is located, nor do they need to be able to connect directly to it.
• Only minimal footprint is required on each of the distributed nodes where the applications reside. A queue manager can run on a fraction of a core.
• Applications can talk to trusted local components. Better from a security, latency and reliability point of view.
• New applications with their own queue managers can be dynamically added to the messaging network and immediately be able to send messages to any queue destination on it.
• The performance and availability properties of an application’s queue manager can be defined by the application team to match with the applications needs.

Let IBM MQ handle transmission across boundaries within your landscape such that your applications can run as truly decoupled components.

Messages are individually transactional

Applications typically think in terms of completing individual business activities – processing a payment, submitting a sales order, booking a flight. They need to be sure that each business activity moves completely from one state to another. The code includes how to handle any errors in a consistent way for each individual transaction.

Messaging grew up in this transaction centric world and the API for putting and getting messages is specifically suited to co-ordinating individual technical updates, potentially even combining messaging with other non-messaging resources in global transactions.

Streaming interfaces such as that of Apache Kafka aren’t designed for individual transnationality. They are optimised for exceedingly high throughput, and so by default do not commit events individually, preferring to send them in batches to optimise the interaction pattern. Whilst you can force their interface to send individual messages, this requires a concerted effort in the way you code and configure both client and server. Key configuration points can be easily missed, and performance will definitely be compromised.

You may recognise the above as a version of the “transactional outbox pattern”. This is a mechanism often used to ensure internal updates to an application are aligned with anything sent beyond its boundary, and indeed this pattern is often used to reliably connect an application to a streaming interface and yet retain exactly once delivery.

So, in summary, it is better to have the applications work in their naturally transactional way with messaging, simplifying their code, and ensuring an immediate, transactionally accurate capture of all events, then the MQ to Apache Kafka connector can handle the transition into the streaming interface.

Messaging only sends what you need where you need it

IBM MQ topics have a few features that are extremely well suited to event delivery across a distributed environment and may be the most efficient way to get just the right amount of data to Kafka.

Perhaps one of the most powerful aspects of IBM MQ’s publish/subscribe feature is that topics enable subscription based on wildcards within the topic string. This means a consumer can subscribe to a very specific subset of messages and only those will be transmitted across the messaging network. It is not possible to do this sort of selective replication between Apache Kafka installations in such a lightweight way as Kafka replicates entire topics.

Topic filtering can significantly reduce the network traffic between the sources of events, and the Kafka installation. This could be important for preservation of bandwidth, or to optimise on ingress/egress costs between cloud providers, to save on storage, or simply to prevent unnecessary processing on the consumer side.

Messages can be individually prioritised

Some messages are simply more important than others from a latency perspective. Their data needs to be acted upon in a shorter timeframe in order for it to be effective.

Take for example road traffic information. A report of an incident would need to be extremely timely and would certainly sit higher in priority than an updated weather report.

If you are using Apache Kafka to perform stream processing for example with Apache Flink, you can use IBM MQ’s messaging priority to ensure that the most time critical events are “seen” by the stream processing algorithms in a timely manner regardless of the wash of lower priority events.

Message persistence is a choice

Messages are only held in the queue until they are delivered, at which point they are deleted. This is one of the defining differences between IBM MQ and Apache Kafka. Kafka stores events in an immutable event log and it is a Kafka administrator’s decision as to when to delete them, rather than the applications.

This difference becomes important in a number of ways:-

• Messages are only held, whether in memory or on disk, whist they are in transmission. They remain indefinitely until they are delivered to their destination. This means that they only take up the storage necessary for in-process events, making storage requirements much simpler. To do the same thing with Apache Kafka would require dramatically more storage at every point in the network.
• Modern high availability techniques in IBM MQ mean that simple storage types can be used, further simplifying system requirements to house a queue and making it easy to deploy on agnostic infrastructure.
• You don’t have to durably persist messages to disk. If your focus is performance and you can tolerate potential loss of some data on failure, then you can configure non-persistent queues, optimising throughput and further simplifying the infrastructure requirements.
• Persistence can be a per message decision. It is possible for the application itself to decide whether a given message should be persisted or not allowing you to define an even more specific balance between performance and data integrity.

This demonstrates that it makes much more sense to use IBM MQ as an asynchronous transport to traverse your landscape and then use Apache Kafka for processing and analytics.

I think we’ve said enough!

We could go on and discuss the subtleties of how IBM MQ messages can be grouped and segmented or how they can optimise transmission across poor networks, or provide message level encryption, but at the end of the day, hopefully the main point is clear. It makes a lot of sense to consider that IBM MQ is one of the best sources of events. Furthermore, there is considerable benefit in using its messaging network to do the leg work of transporting events across the landscape to Apache Kafka.

Acknowledgements

Sincere thanks to Aiden Gallagher, Nick Glowacki and David Ware for their input and review on this article.