You are probably aware that IBM MQ supports both topics and queues as message endpoints. These behave differently in several ways, but one fundamental difference is that topics are used to implement a pub-sub pattern while queues are used for point-to-point. Did you know that topics and queues can be joined to implement new, more varied patterns? One of these offers similar behavior to the streaming queues feature (new in 9.3) and may be of interest to those of us still on prior versions of MQ.
Using this pattern, we can have a producer that publishes to a MQ topic, and then one or more consumers that are reading the messages from one or more queues. The most basic implementation of this pattern looks like the following diagram.
Messages m1, m2 and m3 are published by producer P1 to the topic. Because the queue is defined with a subscription to the topic the queue manager automatically puts a copy of each message to the queue. The three consumers, C1, C2 and C3, each receive messages from the queue as if point-to-point were happening. The configuration looks as follows:
*
* Topic definitions:
DEFINE TOPIC('TOPIC') TOPICSTR('topic-string') DEFPSIST(YES)
*
* Local queue definitions:
DEFINE QLOCAL('QUEUE.V1') DEFPSIST(YES)
*
* Subscriptions:
DEFINE SUB('sub-1') TOPICSTR('topic-string') DESTCLAS(PROVIDED) DEST('QUEUE.V1') PUBPRTY(ASPUB)
Of course, there is no restriction on the number of queues that can subscribe in this way. You can extend the implementation so that each message published is copied once to multiple queues subscribed to the topic, creating a pre-9.3 version of streaming queues. This is a common pattern our team has been using since first adopting IBM MQ over a year ago.
What happens in a uniform cluster?
We recently switched to using MQ uniform clusters, grouping 2-3 queue managers together. The focus is on “uniform.” From a configuration point of view the only difference is the queue manager name. All have identical queue and topic definitions. A client may connect to any queue manager in the cluster, with the goal being to replicate the same experience of publishing to a topic and then having a message copied once to a single queue.
If we use the previous MQ definitions as they are, then we do not get the desired behavior. Assume we have two queue managers in the cluster, QC00 and QC01. P1 connects to QC00. C1 and C3 connect to QC00. However, C2 connects to QC01 as MQ attempts to keep the connections balanced. Messages are being handled, but all the workload is happening from QC00. P1 publishes a message at QC00’s local topic, QC00 copies this to the local queue and then C1 or C3 get the message. C2 is shut out of the picture. Not ideal.
The fix is to make the local queue in each case cluster-aware, as follows. This allows QC00 to balance the messages across the cluster, sending roughly half of those written to its local queue to the same queue on QC01, where C2 is waiting for work. Problem solved.
*
* Topic definitions:
DEFINE TOPIC('TOPIC') TOPICSTR('topic-string') DEFPSIST(YES)
*
* Local queue definitions:
DEFINE QLOCAL('QUEUE.V1') DEFPSIST(YES) CLWLUSEQ(ANY) CLUSTER() DEFBIND(NOTFIXED)
*
* Subscriptions:
DEFINE SUB('sub-1') TOPICSTR('topic-string') DESTCLAS(PROVIDED) DEST('QUEUE.V1') PUBPRTY(ASPUB)
What about clustering the topic also?
If making the queue cluster-aware, what about doing the same for the topic? This is done by adding the “CLUSTER()” option to the topic definition as well. Seems reasonable, but as it turns out, it results in messages being duplicated by the queue managers. Probably not the behavior you want.
The full picture
Let’s round out the picture by summarizing how setting the cluster option on the queue and topic works for each variation. Consider our original diagram in the context of a uniform cluster with queue managers, QC00 and QC01. Assume P1 is connected to QC00. This is what happens to our three messages in each case.
|
Topic clustered
|
Queue clustered
|
Contents of queues
|
Interpretation of results
|
|
QC00
|
QC01
|
|
No
|
Yes
|
m1, m3
|
m2
|
The topic is not clustered and only the QC00 subscriber receives the message. However, because the queues are clustered the messages are divided between the two queues in a round-robin fashion. There is effectively one logical subscription to the topic. Exactly one copy of each message is made and placed to the subscribing queues.
|
|
No
|
No
|
m1, m2, m3
|
empty
|
Without any clustering in place all messages published to the topic hosted at QC00 go only to the subscribed queue at the same queue manager. The queues do not share in the message distribution. Only consumers connected to the queue at QC00 will receive the messages. The topics and queues share a common name, but nothing else.
|
|
Yes
|
Yes
|
m1, m2, m3, m2
|
m1, m3
|
Because the topic is clustered any messages published to the topic on QC00 also appear on the same topic at QC01. This occurs since the goal of publishing to a topic is that every subscriber receives a copy of the message.
Each queue manager is managing a subscription to the topic, and a message is copied to each. As a queue manager places messages onto the local queue these get distributed in the same round-robin manner as observed in the first scenario. (Note, the exact distribution of the messages may not always match the illustration).
|
|
Yes
|
No
|
m1, m2, m3
|
m1, m2, m3
|
The outcome in this case is like the prior one, only in a more predictable manner. The clustered topic again propagates messages throughout the cluster. However, since the queues are not clustered there is no further distribution of the message load. Each queue has copies of the three original messages available for consumption.
|
Conclusions
I hope you found this information useful and insightful. It was a fun afternoon exercise to understand the implications of each scenario. Consider using the topic-to-queue pattern whenever any of the following holds:
- The producer publishes messages to a topic and the messages must be read and processed only once (i.e., as in point-to-point).
- Messages are produced once but are consumed in point-to-point fashion by various consumers (a.k.a. streaming queues in 9.3).
- You want to have messages consumed on queues from multiple levels of the topic tree. In my example, I have shown the subscriptions at the root of the tree but these can be defined at any level.
#ibmchampions-highlights-home#ibmchampions-highlights