MQ Streaming Queues

What are Streaming Queues?

In MQ 9.2.3 we introduced a new feature called Streaming Queues. Streaming Queues are designed to give you a way of having the queue manager take a copy of the messages put to local queues, and deliver the copies to a separate queue or topic of your choosing.

With streaming queues configured, when a putting application puts a message to the original queue a near-identical duplicate message is delivered to the stream queue. They are designed to give you a convenient way of capturing a duplicate stream of messages in order to:

• Streaming messages to Apache Kafka using the Kafka Connect source connector for IBM MQ.
• Performing analysis on the data going through the system.
• Storing messages for recovery at a later time.
• Capturing a set of messages to use in development and test systems.
• Consuming IBM MQ event messages from the system event queues, and sending additional copies to other queues or topics.

Because duplicate messages are delivered to the stream queue at put time, messages already on the original queue when the stream queue is defined are not sent to the duplicate queue.

The messages sent to the stream queue are almost identical with a small number of exceptions. For example, the following aspects of the stream queue messages are identical to the original message:

• Message ID
• Correlation ID
• Message Properties
• Application Data

The following parts of the stream queue message may be different to the original:

• Report options that cause the queue manager to generate new messages are removed
• Expiry is set to unlimited regardless of the expiry of the original message. If the stream queue has a CAPEXPRY value configured then that value will be applied to the streamed message.

Configuration

Configuration is done on a queue-by-queue basis. For example if messages are currently delivered to QUEUE1 and you would like duplicates of those messages streamed to QUEUE2 you could set the new STREAMQ attribute as follows:

ALTER QL(QUEUE1) STREAMQ(QUEUE2) STREAMQOS(BESTEF)

The STREAMQOS attribute allows you to choose between the 2 streaming queue operating modes:

• Best Effort (BESTEF)
  • This option is the default and is the best option for scenarios where you do not want the original application to be affected by the streaming queue feature. Messages will be delivered to the original queue (in our case that’s QUEUE1) regardless of any problems encountered when streaming messages to QUEUE2. So if QUEUE2 is full when a message is put by the application it will be delivered to QUEUE1 for processing even though the queue manager isn’t able to stream a copy to QUEUE2.
  • Set using ALTER QL(QUEUE1) STRMQOS(BESTEF)

• Must Duplicate (MUSTDUP)
  
  • This option is best for scenarios where it is important that each message delivered to the original queue is also delivered to the stream queue. If there is a problem streaming a message to QUEUE2 then the message won’t be delivered to QUEUE1 and the putting application will receive an appropriate error code. The application will need to retry its put just like it would if QUEUE1 had a problem.
  • Set using ALTER QL(QUEUE1) STRMQOS(MUSTDUP)

What if I want multiple copies?

Streaming queues allow you to stream the duplicate messages to alias queues as well as local queues. If you define a queue alias to a topic, and configure a queue to stream messages to that topic alias, the duplicate messages are delivered to any subscribers to that topic based on MQ’s existing publish/subscribe semantics. Importantly the messages delivered to subscribers will not be identical to the original because MQ’s existing publish/subscribe behaviours are retained. One example of this difference is that each message delivered to subscribers will have a new message ID.

It is also worth nothing that if there are no subscribers to the topic the duplicate messages will not be delivered to any applications even when the original queue is set with STRMQOS(MUSTDUP).

How does security work with streaming queues

For putting applications, once a queue has been configured to stream messages to another queue the putting application only needs to have authority to put messages to the original queue. Delivery of the duplicate messages is allowed so long as the application can put messages to the original queue. This means that you don’t need to add any new authority records for your putting applications.

For configuration it is important that a user with change (CHG) authority on QUEUE1 cannot simply configure MQ to stream duplicate messages to other queues on the queue manager. This means that to configure QUEUE1 to stream messages to QUEUE2 the user must either a) be ‘mqm’ or the platform equivalent administrator, or b) have CHG authority on both QUEUE1 and QUEUE2.



If you want to try out Streaming Queues yourself you can find more information in the IBM Docs page. You can also try the demo that's available to download from https://github.com/ibm-messaging/mq-streaming-queues (requires an installation of MQ 9.2.3 or later)