Introduction
From IBM MQ 9.3.1, CAPEXPRY becomes a separate attribute on Distributed platforms, replacing the text based attribute in the CUSTOM field. With this change come several new caveats on how this attribute is used, resolved and validated. This article will focus on the motivation behind this change, the details on using this attribute, and how it will co-exist with the existing CUSTOM CAPEXPRY attribute. By the time you finish reading this article, you will be confident in using this new attribute in your own projects and environments.
So let's begin.
Motivation for Change
The original CAPEXPRY attribute was introduced in MQ 8.0.0.4 and can be used to specify the maximum time, in tenths of a second, until a message becomes eligible for expiry processing. Essentially what that means is that it allows the administrator to set a message expiry without making any specific changes to the application itself.
Prior to MQ 9.3.1, CAPEXPRY was part of the MQSC CUSTOM attribute for Queues and Topics. This was done in MQ 8.0.0.4 for historic reasons related to how our LTS and CD releases used to work. But having CAPEXPRY being part of the CUSTOM attribute is not ideal, primarily because it makes CAPEXPRY tricky to find, fiddly to set, it does not benefit from MQSC autocomplete and validation capabilities. CUSTOM CAPEXPRY also wasn’t a clustered attribute, what had its own limitations.
To tackle the above, in MQ 9.3.1 we decided to make CAPEXPRY a first-class MQSC attribute for Queues and Topics, meaning that it will no longer be part of the CUSTOM attribute and will be a separate, independent attribute by itself. This is what we mean by "first-class".
The CUSTOM CAPEXPRY parameter will remain, but we do recommend and urge our users to adopt the new CAPEXPRY attribute and to migrate to it. There's no specific plans at the moment, but it is possible that in some future release, the CUSTOM CAPEXPRY way of setting the message expiry will be deprecated and even potentially removed - again, there's no specific plans for this yet, but something that we are considering doing at some point. So adopting the new CAPEXPRY parameter early in your scripts and environments could be beneficial.
The new CAPEXPRY attribute applies to all queue types, so QALIAS, QLOCAL, QMODEL and QREMOTE, as well as TOPIC objects. It is worth mentioning that there is no automatic migration of the existing queues and topics objects using CUSTOM CAPEXPRY to use the new CAPEXPRY attribute - this is something that you would have to do manually in your scripts and environments.
The new CAPEXPRY attribute is also a clustered attribute, so you will be able to set it on Clustered Queue and Topics objects.
How does the new CAPEXPRY attribute work - Queues
The new CAPEXPRY attribute can be set to any positive decimal value, excluding zero.
The default value of CAPEXPRY for Queues is NOLIMIT, a keyword that we have introduced to mean that there is no limit on the expiry time of messages.
If CAPEXPRY is set to the default, i.e. it's set to NOLIMIT, the CUSTOM CAPEXPRY attribute value can be set and used. This has been done for backward compatibility, so to ensure that your existing scripts using the CUSTOM attribute continue to work post MQ 9.3.1.
If CAPEXPRY is not set to the default, i.e. it has been explicitly set to some value, then the CUSTOM CAPEXPRY attribute can not be set and can not be used. Trying to do this will result in a AMQ8456: Conflicting parameters error, and the expiry value will not be set.
For a "chain of queues", i.e. for example in a scenario where you're putting a message on a remote queue and getting it from a target queue, the lowest Expiry value in the chain is honoured. This is also applicable to Cluster queues.
Here are some examples of using the new CAPEXPRY attribute with queues.
On the top left, you can see that when we define a local queue, we are using NOLIMIT as the default value for the new CAPEXPRY attribute.
On the top right, we're setting the Expiry value for the Queue via the CAPEXPRY attribute to be 1000 10'ths of a second.
On the bottom left is a compatibility example of setting the CUSTOM CAPEXPRY attribute - do notice that CAPEXPRY is set to the default NOLIMIT value.
If CAPEXPRY in this example was set not to NOLIMIT, but to some value, for example 100, we would get the conflicting parameters error since we would be attempting to set both the new CAPEXPRY attribute and the CUSTOM CAPEXPRY attribute.
Finally notice in the bottom right that existing scripts that use the CUSTOM CAPEXPRY attribute will continue to work. This is because for Queues CAPEXPRY is set to NOLIMIT by default, so the command on the bottom right is essentially no different to the command on the bottom left.
How does the new CAPEXPRY attribute work - Topics
As with Queues, CAPEXPRY can be set to any positive decimal value excluding zero. The default value of CAPEXPRY for Topics is ASPARENT, a keyword that we introduced to mean that message expiry is based on the setting of the closest parent administrative topic object.
If CAPEXPRY is not ASPARENT, i.e. it is not set to Default, the CUSTOM CAPEXPRY attribute can not be set. What's interesting here is the use of the NOLIMIT keyword - for Topics, NOLIMIT as a value is equivalent to a numerical value and is treated as such, unlike for Queues where NOLIMIT is a default value. This in turn means that you can not set a Topic object to have CAPEXPRY set to NOLIMIT and CUSTOM CAPEXPRY set to a value - this will result in the aforementioned parameter conflict error.
The expiry value for a message published to a topic is set to the minimum of the value provided by the publishing application or the value inherited from an administered topic in the topic tree.
The expiry value for a topic is derived from the first non-ASPARENT CAPEXPRY value from a topic object at or above the point of publication in the tree.
Before we move on to some examples, it's important to discuss the special case of BASE topic objects - the SYSTEM.BASE.TOPIC and the SYSTEM.ADMIN.TOPIC. They are special, because their default CAPEXPRY value is not ASPARENT, but is NOLIMIT. There is no policing at all of the CAPEXPRY and CUSTOM CAPEXPRY values, so they can be set together. In a situation where both are set, the first-class CAPEXPRY value takes precedence over the CUSTOM CAPEXPRY value.
If we are traversing the topic tree and we reach the SYSTEM.BASE.TOPIC, and on the way we did not encounter any non-ASPARENT value for the CAPEXPRY parameter, we will look at what the CUSTOM CAPEXPRY parameter is set to for the topic object of interest and resolve from that.
To summarise, the process of Expiry resolution for a topic can be seen in this diagram:
Let’s step through what is happening. Assume that we are publishing with an unlimited message expiry to a topic object. We first consider whether the CAPEXPRY attribute is set, and what it is set to: if it is set to an integer value or to NOLIMIT, we set the message expiry to that value (keep in mind, this is assuming that the expiry value of the topic object is smaller than the expiry value of the incoming message. If it is not smaller, we use the expiry value already in the message. Basically, we use the lower of the two).
If CAPEXPRY is set to ASPARENT, i.e. it's default, we take a look at the Expiry value of the Parent. We continue doing this recursively until we reach an integer or NOLIMIT value for the CAPEXPRY attribute.
If we can't resolve a value and we have reached the SYSTEM.BASE.TOPIC, and it is set to NOLIMIT (so default), we try to resolve the Expiry value starting from the point of publication using the CUSTOM CAPEXPRY attribute. So we repeat the process, we check whether it has been set to an integer or unlimited, if yes - then set Expiry to that, if not - move up and try to resolve recursively from Parent, and so on. If we again reach the SYSTEM.BASE.TOPIC and we have not resolved the Expiry to anything, we will use the Expiry value that was set in the publication message.
Here's some examples, so this becomes crystal clear.
For all these examples, assume that we are publishing to the Child node with an unlimited message expiry.
In the first example, we are coming in with an unlimited message expiry, which is larger than the 15000 10ths of a second that are set via CAPEXPRY on that Child node - so message expiry is resolved to 15000.
In the second example, we are again coming in with an unlimited message expiry, this time the Child's CAPEXPRY is set to ASPARENT, so we go up to the Parent - that is ASPARENT as well, so we go up to the Grandparent - that's set to NOLIMIT, so message expiry will be set to NOLIMIT.
In the third example, we are again coming in with an unlimited message expiry - the Child's CAPEXPY is ASPARENT, Parent’s is ASPARENT and so on, so we reach the SYSTEM.BASE.TOPIC, which is set to a default value of NOLIMIT. Now we check what CUSTOM CAPEXPRY is set to - we return to the node of publication (in our case that's the Child node), CUSTOM CAPEXPRY there is unset i.e. it has a default value of ASPARENT, so we move up to the PARENT - it has a value of 200. So Expiry is resolved to 200.
Hopefully that makes it a little bit more clear on how we will now be resolving Expiry for Topics in this new CAPEXPRY and CUSTOM CAPEXPRY co-existence world.
CAPEXPRY and Clustered Objects
A note on using CAPEXPRY with Clustered objects. As mentioned previously, unlike the original CUSTOM CAPEXPRY, the new CAPEXPRY attribute is clustered. So you can now set CAPEXPRY on clustered queues and topics, and the attribute will behave similar to other clustered attributes such as Persistence and Priority. Similar as to what happens currently with other clustered attributes, it's important to keep the resolved CAPEXPRY value consistent across the cluster. If you ever get into a situation where the CAPEXPRY attribute value is inconsistent across the cluster, you will be warned via a warning message.
Also worth mentioning that since CAPEXPRY has only been implemented currently on Distributed platforms, it will not work in a cluster where the full repositories are on z/OS, primarily because z/OS full repositories will not understand the new CAPEXPRY attribute and will ignore it. Implementation of CAPEXPRY on z/OS is planned to roll out eventually though, so keep an eye out for updates.
Real-world CAPEXPRY use case - Streaming Queues
The Streaming Queues feature for IBM MQ allows you to configure a queue to put a copy of every message incoming to some other second queue of your choice. And this can be very useful in many scenarios where you would need to create a copy of your messages.
Some examples where this could be useful include:
- Streaming messages to Apache Kafka using the Kafka Connect source connector for IBM MQ.
- Performing analysis on the data going through the system.
- Storing messages for recovery at a later time.
- Capturing a set of messages to use in development and test systems.
- Consuming IBM MQ event messages from the system event queues, and sending additional copies to other queues or topics.
If you'd like to find out more about Streaming Queues, do take a look at our IBM Documentation.
One use for streaming queues is to create duplicate messages which will be stored for a short period of time as a contingency measure. CAPEXPRY can be used for managing the depth of your queue that is holding these messages. You could set CAPEXPRY on the streaming queue, so very time a message on that queue reaches and exceeds the CAPEXPRY value, the MQ expiry task will remove this message from the queue. You can use the ExpiryInterval parameter (in the TuningParameters stanza of the qm.ini file) to set how frequently you want the expiry task to run.
Do remember that CAPEXPRY has queue scope, and updating or changing the CAPEXPRY value will only take effect on the new messages that are arriving - the messages that are already on the queue will retain the message expiry values that have been set previously.
Here’s the situation above as a diagram:
In this example, we've got messages arriving at a rate of 10000 per second across 5 queues.
The plot diagram shows the depth of one of the queues, where CAPEXPRY has been set to 60 minutes and the ExpiryInterval was set to 5 minutes.
The expiry task is running every 5 minutes, checking for messages that are 60 minutes or older. The queue will continue to grow until the expiry task will find the expired messages and will start deleting them. This allows us to keep the queue depth at a stable and appropriate to our needs and requirements level, as indicated by the saw-tooth effect on the right hand side of the plot.
Do reach out to us!
As always, the MQ Dev team and I are always happy to chat, we're always interested in finding out your thoughts, concerns, any suggestions and so on.
We're always happy to help, or just have an informal chat.
You can reach me via my work email (vasily.shcherbinin@ibm.com) or via LinkedIn.
#IBMMQ
#9.3 #expiry