MQ

 View Only

Morag's Quirks #16 - Queue File Size

By Morag Hughson posted Mon December 18, 2023 04:58 AM

  
Morag's
Quirks
This is part of an occasional series of small blog posts where I (Morag) will write about some of the quirks in IBM MQ, in the hope of invoking the response, "Well, I didn't know that!" Read other posts in this series.

Recently, we were helping an MQ user who was concerned about the amount of data on a particular queue. We helped to change some settings to purge older messages from this state queue, but oddly he said it didn't make any difference - the queue was still just as sizable.

After a few more questions, we realised that he was looking at the size of the queue file on the file system, whereas we were thinking about the number of messages on the queue.

Long story short, the size of the queue file does not immediately shrink when messages are removed from the queue. While one might argue that this is sensible and not quirky, I think it is clear that it is an unexpected behaviour that trips up users and from that perspective it is worth writing about.

On a distributed queue manager, each local queue has a queue file behind the scenes containing the message data. When more messages are put onto the queue, the queue file will increase in size as required to accommodate the data.

You can either look at the file system in the mqmdata/qmgrs/qmname/queues folder, or since MQ V9.1.5, the queue file size is presented as one of the values shown with DISPLAY QSTATUS(q-name), the CURFSIZE attribute. This number is rounded up to the nearest megabyte, and so the smallest reported queue file size is 1MB. If you need to know a more accurate file size, you have to look in the file system. For most people a granularity of a megabyte is plenty.

In the examples below, I'm ignoring the effects of the queue buffer which is probably having a small impact on the numbers.

Example 1

I put five thousand 100 byte persistent messages onto my queue.

                     Current
                    ---------
Messages                 5000
Message Bytes      488.28 KBs
Data Bytes           2.21 MBs
Queue File Size         5 MBs

So I have put 5000 * 100 bytes of data onto my queue, a little under 500 KBs. However each of these 5000 messages also has an MQMD associated with it, which is another 364 bytes of data per message, which is where we get to the calculation that there are over 2MBs of data to be stored. The queue file size is somewhat bigger at 5MBs because of the inefficiency of storing very small messages in a file with a block size of 512 bytes.

If I remove all the messages off the queue, the queue file size remains at 5MBs for a while.

Point of interest: if I remove all the messages off the queue with MQSC command CLEAR QLOCAL(q-name) (or it's PCF equivalent), then the queue file size is immediately reduced.

According to IBM Docs, unused space in a queue file is only released in the following circumstances:

  • When no applications have the queue open
  • After 1000 writes to the queue manager log, or
  • At queue manager shutdown

From experimentation, it is clear that the first two bullet points above must be taken together, i.e. that both must be true, but also that the number appears to be considerably larger than 1000. Experimentation suggests closer to 12,000.

Example 2

I put one hundred 5000 byte persistent messages onto my queue. This is the same amount of data as my first example, a little under 500 KBs, spread across fewer messages. This means the total amount of data contributed by the MQMDs is much smaller as there are only 100 of them. This, as we can see below, fits into a queue file still only at 1MB (or less) in size.

                     Current
                    ---------
Messages                  100
Message Bytes      488.28 KBs
Data Bytes         523.83 KBs
Queue File Size          1 MB

Example 3

To finish off with a third example, I've changed my empty queue to have the maximum possible file size of 100TB (I don't intend to fill it!) which means instead of a block size of 512 bytes, the file will have a block size of 64KB.

ALTER QLOCAL(q-name) MAXFSIZE(104857600)

Now I put five thousand 100 byte persistent messages onto my queue - exactly as I did in the first example. This is a deliberate worse possible case. Really small messages being stored in a queue file with the largest possible block size. The totals shown below differ from the first example only on the queue file size, and this is because of the block size of the file.

                     Current
                    ---------
Messages                 5000
Message Bytes      488.28 KBs
Data Bytes           2.21 MBs
Queue File Size       158 MBs

This shows that you should not increase the maximum queue file size without some thought to the size of the messages that will be stored on your queue. Knowing the average size of your messages would help with that thought process of course.

You can get an instant perspective on the average size of messages on your queue using the QLOAD summary display, or by collecting queue statistics. The latter of course will give you a broader perspective rather than just a snapshot in a point in time.

Summary

So remember, there is a queue file containing the data on your queue and it will grow when needed and shrink more lazily. Don't expect it to immediately dispose of the unused space in the file.

The tables shown throughout this post are from a recent update to the QLOAD summary display which you can read more about here.

Here are some resources for further reading about controlling the size of your queue files:

Tape measuring workman image by Nsit0108.

Morag Hughson is an MQ expert. She spent 18 years in the MQ Devt organisation before taking on her current job writing MQ Technical education courses with MQGem. She also blogs for MQGem. You can connect with her here on IMWUC or on Twitter and LinkedIn.


#mquirks
#IBMMQ
#IBMChampion

2 comments
37 views

Permalink

Comments

Morag Hughson
Thu December 21, 2023 12:02 AM

Hi @Andrew Hickson,

I am very happy to have your expertise here to help enlighten all readers here. I had noticed that the number of log writes was well beyond 1000. I got a number around 12,000, but that was on a development queue manager on a laptop, with default log sizes, so likely downgraded as you mention.

I have asked Hursley to check on the referred to doc content, so hopefully it will get an update, and I will make an update to this blog post in line with that. I suggested that CLEAR QLOCAL might be useful to add to the list there.

It is interesting to learn that not all shutdown would clear space, but understandable that an abrupt termination might not do so. Perhaps it should say "At a normal queue manager shutdown".

Have you ever considered writing some posts on here? Your expertise would make for articles that would be very enjoyable to read.

Hope you are keeping well.

Merry Christmas,
Cheers,
Morag

Andrew Hickson
Wed December 20, 2023 12:45 PM

Morag correctly quotes the IBM doc on this subject, unfortunately I suspect the doc is somewhat out of date in this respect. As always, my knowledge of the code dates to when I retired from IBM and someone in the current development team might provide a more current perspective.

Queue housekeeping is (or was when I was involved!) typically performed as part of a queue manager checkpoint (where the content of the queue file and the recovery log are consolidated). In the early days of MQ checkpoints typically occurred every 1000 log writes, which I expect is where the reference to 1000  log writes originated.

A modern queue manager can easily do tens of thousands or persistent messages per second, at which point doing a checkpoint every 1000 log records becomes a bit onerous. Many years ago I changed this to every 10,000 log records, and IIRC I further updated it to default to every 50,000 records in more recent times (perhaps in the V8 timeframe). The 50,000 log records trigger would be overrridden if log space was insufficient to cope with 50,000 log records between checkpoints. 

Lastly, not all queue manager shutdown's would result in queue compaction (where queue file space is released), for example if the target time for shutdown were exceeded and the shutdown was upgraded to a more abrupt termination.

Similarly it was too restrictive that the queue file associated with an open queue was never truncated and so over the years a number of limited changes were made to look for opportunities to reduce the queue file size of large queues (limited resources were applied to this so it's still very simplistic and restrictive).