This is part of an occasional series of small blog posts where I (Morag) will write about some of the quirks in IBM MQ, in the hope of invoking the response, "Well, I didn't know that!" Read other posts in this series.

The quirkiness of the feature I want to write about today catches people out because it works differently when connected locally than it does when connected over a client. Many times users will try something out locally first and find it does what they need, and then will change to using a client when deployed 'for real' and be disappointed to discover it doesn't work the way they expected from their initial tests. This is one of those and perhaps just writing something that describes this difference may help future users.

Automatic Queue Manager Segmentation

I'm writing about Queue Manager segmentation which happens when an application puts a message and uses the Message Descriptor field MsgFlags to say MQMF_SEGMENTATION_ALLOWED. This tells the Queue Manager that, if it needs to, it is allowed to split up this message into a series of smaller messages in order to fit them on the queue.

The sending application must give the go-ahead for this to happen because the recipient application must also use an appropriate MQGET option to reassemble the segments.

Putting Application

mqmd.Version    = MQMD_VERSION_2;
mqmd.MsgFlags = MQMF_SEGMENTATION_ALLOWED;
memcpy(mqmd.GroupId, MQGI_NONE, MQ_GROUP_ID_LENGTH);
MQPUT(hConn,
      hObj,
      &mqmd,
      &PutOpts,
      MsgLen,
      pBuffer,
      &CompCode,
      &Reason);

Getting Application

GetOpts.Options |= MQGMO_COMPLETE_MSG;
MQGET(hConn,
      hObj,
      &mqmd,
      &GetOpts,
      BufferLen,
      pBuffer,
      &DataLen,
      &CompCode,
      &Reason);

Example of Using Segmentation

Before I get to the quirk that catches people out, let's look at a use of this feature. I have a queue with a maximum message size of 10K.

DEFINE QLOCAL(APP.QUEUE) MAXMSGL(10240)

My putting application, however, has a need to send the getter a 40K message. So the putting application uses code like that shown above and makes a message in the put buffer of 40K, and provides the message length on the put to also show that the message in the buffer is 40K in length.

If you want to try this out for yourself, the Q program can be used with the following invocation, where -as is the message attribute flag for 'segmentation allowed', and -M#x1/40960 puts a single message of size 40K containing all the hex values as its data.

q -m MQG1 -o APP.QUEUE -as -M#x1/40960

The Queue Manager will accept this, and if you view the queue, you will see that it now holds 4 messages, each of 10K in length, and each starting at a given offset of the original message buffer.

A getting application can retrieve this message, reassembled, by using the MQGMO_COMPLETE_MSG option shown in the code snippet above.

If you want to try this out for yourself, the Q program can be used with the following invocation, where -ac is the message attribute flag for 'complete message', -dld2 are display options to show the message length and the MQMD at detail level 2 which shows the MsgFlags, and -q is quiet mode so the large message is not printed on your screen.

q -m MQG1 -i APP.QUEUE -ac -dld2 -q

This will also work over a client connection, so long as the server-connection channel is defined with a maximum message length big enough for the whole message.

ALTER CHANNEL(MQGEM.SVRCONN) CHLTYPE(SVRCONN) MAXMSGL(40960)

The default maximum message length on a server-connection channel is 4MB, so for a small message like this, it is not going to be a problem. However, it will be an important factor later.

What's Quirky About This?

Now let's look at what gets this feature a quirky label.

Imagine a locally bound application that decides to try to use segmentation to put a message bigger than 100MB.

The queue is defined to have the biggest possible message size.

DEFINE QLOCAL(BIG.QUEUE) MAXMSGL(104857600)

As is the queue manager itself.

ALTER QMGR MAXMSGL(104857600)

The putting application creates a message that is 300MB in length and uses the MQMF_SEGMENTATION_ALLOWED flag.

If you want to try this out for yourself, the Q program can be used with the following invocation:

q -m MQG1 -o BIG.QUEUE -as -M#x1/314572800

The message is accepted and when the queue is viewed, there are 4 messages there. It is also interesting (and perhaps quirky) to see that the message is not split exactly on the 100MB mark. But that's a subject for another day and another post.

The getting application can retrieve them reassembled as a single 300MB message with a single MQGET call.

Philosophical Aside: Does this mean that the maximum message size is not restricted to 100MB? The messages on the queue were limited to 100MB, but from the applications perspective they just put and got a single 300MB message. When does a message become a message? Is it when the application puts it? Is it when the queue holds it? Is it when the application gets it?

Now let's try that again over a client connection.

Our server-connection channel is defined to have the biggest possible message size.

ALTER CHANNEL(MQGEM.SVRCONN) CHLTYPE(SVRCONN) MAXMSGL(104857600)

The putting application again creates a message that is 300MB in length and uses the MQMF_SEGMENTATION_ALLOWED flag.

If you want to try this out for yourself, the Q program can be used with the following invocation, where -l mqic tells it to use the client libraries:

q -m MQG1 -l mqic -o BIG.QUEUE -as -M#x1/314572800

This will result in a failure reason code of MQRC_DATA_LENGTH_ERROR (2010)

The message cannot be delivered over the channel to the queue manager because it is too big for the channel maximum message length. The segmentation into smaller messages does not happen until the large buffer reaches the queue manager.

Now it is possible to do application segmentation, and break up the large buffer you need to send into messages that are able to go over the channel, but perhaps you should just design the application to use multiple messages anyway at that point.

If your putting application was a local application, and so able to avoid this restriction, can a client application retrieve the message? Well, using an MQGET, as per the above code snippet, the same restriction applies.

If you want to try this out for yourself, the Q program can be used with the following invocation, where -l mqic tells it to use the client libraries:

q -m MQG1 -l mqic -i BIG.QUEUE -ac -dld2 -q

This will result in a failure reason code of MQRC_DATA_LENGTH_ERROR (2010)

Using MQCB to consume messages

While writing this post, I noticed that this IBM Docs page mentioned that maximum message length on the channel definition does not limit a client application using MQCB rather than MQGET to consume messages.

So I tried it. Unfortunately the Q program does not have a mode to make use of MQCB, and there are no samples that combine MQCB and an option for MQGMO_COMPLETE_MSG, so to test this I made use of the API Exerciser in MO71 - that's what it's for after all!

• First I made an MQCONNX call, filling in the MQCD with my connection details.
• I did an MQOPEN to open my BIG.QUEUE for input.
• I made an MQCB call to MQOP_REGISTER my callback
  • selecting the call back function
  • leaving the MQCBD Max Message Length field at its default -1 value
  • selecting the MQMD version as MQMD_VERSION_2
  • choosing MQGMO Options of MQGMO_COMPLETE_MSG and MQGMO_NO_SYNCPOINT.
• Then I did an MQCTL with Operation MQOP_START.

And sure enough, my client connection, callback was driven with the single 300MB message.

So, the quirkiness of this feature comes from the fact that you can do something with a local connection that you are unable to do with a client connection. Don't get caught out by this.

It is worth also saying here that if you need to send large amounts of data, it's perhaps worth looking at the Managed File Transfer feature of IBM MQ.

Another user raised an MQ Idea (aka Request For Enhancement) for this very behaviour, requesting that parity between the client and local bindings applications be maintained. If you'd also like to vote on this, please read it here.

#mquirks
#IBMMQ
#IBMChampion