MQ

This is part of a series of blog posts which will cover features of IBM MQ that I happen to be using this week. I spend a lot of time using MQ, and not everything counts as an MQ Little Gem, or a Quirk, but is still interesting to hear about. Read other posts in this series.

There are a number of different ways you might set up a network of queue managers to route messages, and sometimes it would be good to be able to test the setup is correct before you let your own applications loose on the system.

IBM MQ has a feature called trace-route messaging that allows the route a message takes through queue managers, queues and channels to be tracked and reported upon.

For example, if you have a route that takes a double hop through one queue manager before reaching the final destination on the second hop, you can discover the route the message takes, landing first on one transmission queue, and then another before arriving at its final destination. In a double hop, the message is sent and received by two different channels, and you get to see their names and types in the information collected as well.

To be able to access the information about the route a message takes through the network, there are a few different options.

1. Inject a special trace-route message into the network, which will accumulate information about where it has been as it goes along, and then when it arrives at the final destination it can either:-
   1. deliver the message to the target queue with all the accumulated route information
   2. discard the message just before putting it to the target queue, and send a reply message back with all the accumulated route information
2. Use a report option on your own application messages, MQRO_ACTIVITY, which will send back report messages to the replyToQ about each step along the way.

You might be wondering about the option to discard the message just before putting it to the target queue. This allows for you to test the routing of messages without impacting any applications that might be running in the system who, upon receiving a trace route message full of accumulated routing information, would just not know what to do with it. To avoid any issues, you can choose to get the information back in a reply message and the message will never actually be put onto the final target queue.

A special trace-route message

While this may sound rather weird, it's not really. It's just a PCF format message, with a particular value, MQCMD_TRACE_ROUTE, in the MQCFH header. A message with this shape is internally recognised by the channels and they add all the information about their gets, sends, receives, and puts, to the message, as the message travels across them.

Included in the information about where the message went, are times about when each operation happened. For example, the time of the get from the transmission queue, or the time of the receive from the network. From these time stamps, overall response times of the message going through the system can also be calculated.

Report Options

As an alternative to using the special trace-route message, you can instead switch on activity reports on your own messages, or any simple message, by using the MQRO_ACTIVITY flag in the Message Description Report field. This will then generate a report message sent back to your ReplyToQ, for each activity the channels do.

An activity may be made up of a number of operations. For example a sender channel does an activity on a message, and reports a get operation and a send operation as part of that activity.

Here's an example.

ACTIVITY
     Application Name::\mqm9300\bin64\runmqchl.exe
     Application Type:11
     Application Description:Sending Message Channel Agent
          OPERATION
               Operation Type:GET
               Operation Date:20230116
               Operation Time:03542189
               Queue Manager:MQG1
               Queue:MQG2
               Resolved Queue:MQG2
               Channel Name:MQG1.TO.MQG2
               Channel Type:SENDER
          OPERATION
               Operation Type:SEND
               Operation Date:20230116
               Operation Time:03542189
               Queue Manager:MQG1
               Channel Name:MQG1.TO.MQG2
               Channel Type:SENDER
          TRACEROUTE
               Recorded Activities:1
               Unrecorded Activities:0
               Discontinuity Count:0

There is an advantage using report messages for this rather than accumulating all the information in the message, because you get reports sent back as the message moves along, so if it gets stuck somewhere, say on a transmission queue of a stopped channel, you can see how far it got along the route from the most recent report message that was returned. This gives you a starting point for your investigation of why the message didn't get all the way through.

Things you can use Trace-route Messaging for

Here are some ideas of the sorts of things you might use trace-route messaging to check.

We already saw the first example of a multi-hop route through several transmission queues and several channels.

Cluster distribution

One of the classic examples quoted where trace-route messaging comes in very handy is checking the distribution across a cluster. You might be looking to check for even distribution across all the target queues, as in this diagram.

Or you might have weighting applied to the various queue manager's in your cluster, so that more of the messages go to one instance than to the others, such as this diagram where I have one particularly heavyweight queue manager that is given more of the messages than the others. By the values I have defined for CLUSWGHT on the channels, I expect the heavyweight queue manager to get 6 times as many messages to process as any of the other queue managers.

Transmission Queues

One of the simple things about a classic clustering set up was that the transmission queue was always SYSTEM.CLUSTER.TRANSMIT.QUEUE, however, there were downsides to that setup so in more recent releases, features were added to MQ's queue manager clustering to allow each channel, or set of channels, to make use of specific transmission queues. While this removes some of the downsides, it does mean that it is harder to know what queues are being used. However, the transmission queue is part of the data recorded in trace-route messaging, so it is possible to see exactly which transmission queues are in use as in this diagram.

More complex setups

There are infinitely many possibilities of how complex your setups may be that you want to discover and/or track using trace-route messaging. You might have overlapping clusters with gateway queue managers, where one gateway is supposed to be used in preference to the other. Trace-route messaging would show you whether either or both of the gateways were being used in the routes.

Creating trace-route messaging data

There are a variety of tools out there to help you to create and view IBM MQ's trace-route messaging data.

Command Line tools

IBM MQ comes with a command line tool, dspmqrte, which will inject a special trace-route message into your network and collect up the replies. It has a plethora of options to allow you to configure the way it operates, including all the mechanisms mentioned above. There is an excellent guide to this tool that was produced in the WebSphere MQ V6 timeframe when trace-route messaging was created, by the author of the feature, that you can read here. I recommend it if you intend to use the tool.

Here's an example dspmqrte command to trace the route through the above overlapping cluster.

dspmqrte -m EXTQM1 -q APP.Q -v outline -o

Mark Taylor has written another version of dspmqrte, named dspmqrtj, which outputs it's data in JSON format. You can read about it here.

Graphical tools

The MQ Explorer has a plug-in written by Mark Taylor, MS0P, which he mentions in the above blog post. This plug-in provided the capability to drive trace-route messaging from a GUI environment, and see a pie-chart of cluster distribution.

MO71 has a Trace Message dialog which also allows you to drive trace-route messaging from a GUI environment, and see a pie-chart of cluster distribution. It provides both a textual and a pictorial version of the output to aid your understanding of the data collected. It is the reason I decided to write this post, because I spent the last week making extensive use of MQ's trace-route messaging because of a recent update to the MO71 Trace Message dialog, which you can read about here.