Morag's
Quirks
This is part of an occasional series of small blog posts where I (Morag) will write about some of the quirks in IBM MQ, in the hope of invoking the response, "Well, I didn't know that!" Read other posts in this series.
Recently, we were helping an MQ user who was concerned about the amount of data on a particular queue. We helped to change some settings to purge older messages from this state queue, but oddly he said it didn't make any difference - the queue was still just as sizable.
After a few more questions, we realised that he was looking at the size of the queue file on the file system, whereas we were thinking about the number of messages on the queue.
Long story short, the size of the queue file does not immediately shrink when messages are removed from the queue. While one might argue that this is sensible and not quirky, I think it is clear that it is an unexpected behaviour that trips up users and from that perspective it is worth writing about.
On a distributed queue manager, each local queue has a queue file behind the scenes containing the message data. When more messages are put onto the queue, the queue file will increase in size as required to accommodate the data.
You can either look at the file system in the mqmdata/qmgrs/qmname/queues folder, or since MQ V9.1.5, the queue file size is presented as one of the values shown with DISPLAY QSTATUS(q-name), the CURFSIZE attribute. This number is rounded up to the nearest megabyte, and so the smallest reported queue file size is 1MB. If you need to know a more accurate file size, you have to look in the file system. For most people a granularity of a megabyte is plenty.
In the examples below, I'm ignoring the effects of the queue buffer which is probably having a small impact on the numbers.
Example 1
I put five thousand 100 byte persistent messages onto my queue.
Current
---------
Messages 5000
Message Bytes 488.28 KBs
Data Bytes 2.21 MBs
Queue File Size 5 MBs
So I have put 5000 * 100 bytes of data onto my queue, a little under 500 KBs. However each of these 5000 messages also has an MQMD associated with it, which is another 364 bytes of data per message, which is where we get to the calculation that there are over 2MBs of data to be stored. The queue file size is somewhat bigger at 5MBs because of the inefficiency of storing very small messages in a file with a block size of 512 bytes.
If I remove all the messages off the queue, the queue file size remains at 5MBs for a while.
Point of interest: if I remove all the messages off the queue with MQSC command CLEAR QLOCAL(q-name) (or it's PCF equivalent), then the queue file size is immediately reduced.
According to IBM Docs, unused space in a queue file is only released in the following circumstances:
- When no applications have the queue open
- After 1000 writes to the queue manager log, or
- At queue manager shutdown
From experimentation, it is clear that the first two bullet points above must be taken together, i.e. that both must be true, but also that the number appears to be considerably larger than 1000. Experimentation suggests closer to 12,000.
Example 2
I put one hundred 5000 byte persistent messages onto my queue. This is the same amount of data as my first example, a little under 500 KBs, spread across fewer messages. This means the total amount of data contributed by the MQMDs is much smaller as there are only 100 of them. This, as we can see below, fits into a queue file still only at 1MB (or less) in size.
Current
---------
Messages 100
Message Bytes 488.28 KBs
Data Bytes 523.83 KBs
Queue File Size 1 MB
Example 3
To finish off with a third example, I've changed my empty queue to have the maximum possible file size of 100TB (I don't intend to fill it!) which means instead of a block size of 512 bytes, the file will have a block size of 64KB.
ALTER QLOCAL(q-name) MAXFSIZE(104857600)
Now I put five thousand 100 byte persistent messages onto my queue - exactly as I did in the first example. This is a deliberate worse possible case. Really small messages being stored in a queue file with the largest possible block size. The totals shown below differ from the first example only on the queue file size, and this is because of the block size of the file.
Current
---------
Messages 5000
Message Bytes 488.28 KBs
Data Bytes 2.21 MBs
Queue File Size 158 MBs
This shows that you should not increase the maximum queue file size without some thought to the size of the messages that will be stored on your queue. Knowing the average size of your messages would help with that thought process of course.
You can get an instant perspective on the average size of messages on your queue using the QLOAD summary display, or by collecting queue statistics. The latter of course will give you a broader perspective rather than just a snapshot in a point in time.
Summary
So remember, there is a queue file containing the data on your queue and it will grow when needed and shrink more lazily. Don't expect it to immediately dispose of the unused space in the file.
The tables shown throughout this post are from a recent update to the QLOAD summary display which you can read more about here.
Here are some resources for further reading about controlling the size of your queue files:
Tape measuring workman image by Nsit0108.
#mquirks
#IBMMQ
#IBMChampion