Streaming Queues and Message Expiry

One use for streaming queues is to create duplicate messages which will be stored for a short period of time as a contingency measure. In this case the CAPEXPRY custom property is set on the streaming queue to  set a time limit on messages. MQ schedules an expiry task run at an interval specified in the mq.ini file (default 5 minutes). This task deletes any messages that are older than the CAPEXPRY value set on it (CAPEXPRY has queue scope, and only affects messages arriving after it has been set, similarly, amending the CAPEXPRY value will only affect subsequent messages).

If the messages on a streaming queue with CAPEXPRY set are not being consumed, then the queue will grow until the expiry task is scheduled and finds messages that are old enough to be deleted.  E.g., the plot below shows the queue depth of a streaming queue.


In this example, messages are arriving at a rate of 10,000/sec across 5 queues. The plot shows the depth of one of the queues, where CAPEXPRY is set to 60mins and ExpiryInterval (the qm.ini file parameter which determines the frequency that the expiry task is scheduled) is set to 5minutes. In this case the queue grows to a until the the expiry task encounters messages that are 60mins or more old. At that point, each time the expiry task is run it will find another 5minutes worth of messages that can be deleted, creating the saw-tooth effect above.

When using message expiry with streaming queues you should consider the depth that the queues will grow to and the additional file system storage that may be required. These considerations and more are discussed in the updated Streaming Queues Performance report on the MQ performance GitHub page : MQ V9.2.3 Streaming Queues Performance Report V1.1