IBM MQ

Easily controllable queue-file sizes

By Louis Horsley posted Wed April 08, 2020 03:50 AM

  
As of MQ 9.1.5 we have introduced new capabilities relating to queue file sizes to increase the maximum size of an individual queue file and limit the maximum size of  queue files. Prior to MQ 9.1.5 on distributed platforms, the only queue attributes available to influence the maximum size a queue file can take up on disk were the following:

  • MAXDEPTH      : maximum number of messages on the queue
  • MAXMSGL        : maximum size of an individual message body
  • MAXPROPL      : maximum number of bytes of property data in a message

However due to possible varying quantities and sizes of messages it was not possible to adequately guarantee the size a queue file may grow to. Prior to MQ 9.1.5 the maximum queue file size is constrained to ~2TB, however as of MQ 9.1.5 we can now dynamically manage the maximum size of an existing individual queue file with queue and qstatus attributes and specify sizes up to 255TB.

For local and model queues we have introduced the queue attribute MAXFSIZE which is the maximum size a queue’s queue file can grow to. Valid integer parameters in multiples of 1MB are from 20 to 267386880 giving us a range of 20MB to 255TB, defining an invalid value will result in an 'AMQ8425E: Attribute value error'. There is also a  parameter constant 'MAXFSIZE(DEFAULT)' which resolves to 2097152 resulting in a default maximum queue file size of 2TB.

Queue files store message data in fixed size blocks (BLKSIZE), which is calculated automatically in relation to MAXFSIZE. Block sizes range from 512B for MAXFSIZE(20) to 64KB for MAXFSIZE(267386880) and are in multiples of 512. The default block size for queue files with MAXFSIZE set to DEFAULT or lower is 512 bytes. It is possible however to set a custom block size as shown in the examples section below. 

In the event that a change to BLKSIZE is requested (whether defined explicitly or automatically calculated) on an existing queue populated with messages, there are two conditions for the new value to be initiated, firstly the queue has to be emptied, secondly the queue has to have received a message. In this scenario a new warning message 'AMQ7493 Granularity Changed' is written to indicate to the user that the queue must be emptied or cleared in order for the new block size to take effect. This may not be a non trivial operation taking a significant amount of time, in this case you may want to periodically check if the operation has completed by comparing the current maximum queue file size to the newly proposed MAXFSIZE value for equality, from MQ 9.1.5 we can do just that with the new queue status attribute CURMAXFS.  

We have also introduced another qstatus attribute CURFSIZE to indicate the current size of the queue file on disk rounded up in megabytes. Be aware that CURFSIZE will track the actual file size on disk as the number of messages increases, however as the number of messages reduce there may be periods where CURFSIZE has not reduced concurrently, this is as expected as MQ has queue compaction algorithms that are optimised for I/O efficiency. 


Demo

  • Create a queue manager:
  • crtmqm QM1 && strmqm QM1

  • Define a queue:
  • echo "DEFINE QLOCAL(Q1)" | runmqsc QM1

  • Check MAXFSIZE:
  • echo "DISPLAY QLOCAL(Q1) MAXFSIZE" | runmqsc QM1

  • You will see the constant value 'DEFAULT', now increase MAXFSIZE to a size that will require a new BLKSIZE:
  • echo "ALTER QLOCAL(Q1) MAXFSIZE(3145728)" | runmqsc QM1

  • Check MAXFSIZE as above, you will see our new value '3145728', now check CURMAXFS:
  • echo "DISPLAY QSTATUS(Q1) CURMAXFS" | runmqsc QM1

  • Instead of our new value we see the default value '2088960'. Remember that even though our queue is empty, we still need a message to have been received, now put a message on the queue:
  • amqsput Q1 QM1

  • Check CURMAXFS again, you will now see the value '3145728', now increase MAXFSIZE to a size that will NOT require a new BLKSIZE:
  • echo "ALTER QLOCAL(Q1) MAXFSIZE(3145729)" | runmqsc QM1

  • Check CURMAXFS again, you will see the value '3145729', we have not had to drain the queue as the increase does not require a new BLKSIZE now increase MAXFSIZE to a size that will require a new BLKSIZE:
  • echo "ALTER QLOCAL(Q1) MAXFSIZE(5242880)" | runmqsc QM1

  • Check CURMAXFS again, you will see a value smaller than '5242880', we will need to drain the queue and put a new message:
  • echo "CLEAR QLOCAL(Q1)" | runmqsc QM1
  • amqsput Q1 QM1

  • Check CURMAXFS again, You will now see our new value '5242880'.

MQSC examples


Define a new queue with a maximum queue file size of 100GB:

DEFINE QL(Q1) MAXFSIZE(102400)
     1: DEFINE QL(Q1) MAXFSIZE(102400)
AMQ8006I: IBM MQ queue created.

Define a huge queue with a maximum queue file size of 100TB:

DEFINE QL(Q1) MAXFSIZE(104857600)
     1: DEFINE QL(Q1) MAXFSIZE(104857600)
AMQ8006I: IBM MQ queue created.

Define a queue with a default maximum queue file size of 2TB and a BLKSIZE of 1024:

DEFINE QLOCAL(Q1) CUSTOM('BLKSIZE(1024)')
      1 : DEFINE QLOCAL(Q1) CUSTOM('BLKSIZE(1024)')
AMQ8006I: IBM MQ queue created.

Reduce the maximum size of a given queue file to the lowest upper bound of 20MB:

ALTER QLOCAL(Q1) MAXFSIZE(20)
      1 : ALTER QLOCAL(Q1) MAXFSIZE(20)
AMQ8008I: IBM MQ queue changed.

Increase the maximum size of a given queue file to the highest upper bound of 255TB:

ALTER QLOCAL(Q1) MAXFSIZE(267386880)
      1 : ALTER QLOCAL(Q1) MAXFSIZE(267386880)
AMQ8008I: IBM MQ queue changed.

Query the maximum size a given queue file can grow to:

DISPLAY QLOCAL(Q1) MAXFSIZE
      1 : DISPLAY QLOCAL(Q1) MAXFSIZE
AMQ8409I: Display Queue details.
      QUEUE(Q1)                           TYPE(QLOCAL)
      MAXFSIZE(DEFAULT)

Query the current maximum file size the queue can grow to and the current size of the file on disk.

DISPLAY QSTATUS(Q1) CURMAXFS CURFSIZE
      1 : DISPLAY QSTATUS(Q1) CURMAXFS CURFSIZE
AMQ8450I: Display queue status details.
      QUEUE(Q1)                          TYPE(QUEUE)
      CURDEPTH(3)                        CURFSIZE(1)
      CURMAXFS(2088960)

Define MAXFSIZE with an invalid value:

DEFINE QLOCAL(Q1) MAXFSIZE(19)
      1 : DEFINE QLOCAL(Q1) MAXFSIZE(19)
AMQ8425E: Attribute value error.
0 comments
20 views

Permalink