MQ

As with all of my posts these days, my knowledge is rather out of date. 

There were VERY significant performance improvements to linear logging when reusable log extents were introduced, and much of this improvement comes from eliminating the need to format the new extents. The queue manager should be monitoring the log usage over 'time' to tune the overall number of log extents.  In a traditional qmgr environment it is advised to have a seperate file system for the recovery log and hence a conservative approach to reducing the number of reusable log extents made sense as little is actually gained by freeing unused file space in a file system dedicated to this purpose.

IIRC the appliance didn't follow this general configuration advice (one file system for both recovery logs and queue manager data).  Despite this I don't recall a different approach to managing log extents in that environment.  RDQM was originally only supported on the appliance, which in turn only supported circular logging. I don't know how the appliance file system configuration has since been ported to a more generic Linux RDQM QMgr (after my time!), and hence whether the original design/configuration considerations would/should need revisiting in that environment.

If you do have a file system configured explicitly for use for MQ recovery log files, then why are you interested in how many reusable log extents exist ?

On the other hand, if an RDQM QMgr does require the use of a single file system for both QMgr data, and recovery logs, then a more dynamic approach to sizing the pool of reusable log extents might make more sense.

I'd also advocate using the "RESET QMGR TYPE(REDUCELOG)" in preference to manually deleting any MQ recovery log files. Manually deleting MQ recovery log files is best avoided unless at the explicit request of IBM MQ support.

As Andrew said, if you want to delete the R type log files, please use the MQSC command

RESET QMGR TYPE(REDUCELOG) REDUCE(AUTO | MAX | ONE)

In your case, you should probably use MAX... and remember that the qmgr normally chooses when an eligible log will be removed, when created with -lla...

I should also have pointed out the obvious, that looking at the log file space usage at a single point in time isn't very meaningful.

When you say "automatic linear logging", is it both the management of the recovery logs AND the management of the media imaging that has been set to be Qmgr managed ?

If you're taking manual images, then immediately after taking a full set of media images there would be expected to be a significant number of R prefixed files. These are the files the qmgr hopes to reuse before the next media images are taken. When using Qmgr internally managed media image recording then the media recovery point would typically be expected to be a little more gradual, and the number of R files would then be likely to be a bit lower (offset by a larger number of S files). You can think of the R files being used in a similar manner to the S files in a circular logging environment. The queue manager hopes to continually write to a much bigger ring of log files and thus avoid formatting any new log files. In the case of circular logging the size of the ring of log files needed relates to the amount of work between checkpoints (and possibly long running transactions). In the case of QMgr managed linear logging the size of the ring is typically related to the amount of work between complete media images of all objects.

Just for others to follow why this is an issue...

In a non-RDQM world you give ALL the QM on a machine a large disk and let them fight out who uses it. Hoping that you have allocated enough disk based on the number of QM,  transactions, message size etc. etc. If you screw up the sizings you can simply increase the size of the disk and you're all good. 

RDQM has this horrible issue where you have to know up-front what each QM (not machine) is going to use. If you get it wrong the individual QM could blow up (hopefully not corrupting itself) then you have to delete the QM and recreate it with more disk allocation. This is not clever enough IMO - you should be able to just increase the volume allocated to the QM. However,  it is what it is and I'm not about to worry about which piece of software has the limitation or why (presumably DRBD).

However, having raised a PMR for this I found out about a non-documented qm.ini param MaxReusableLogExtents. I can set that value and have some warm feeling that I'm safer than I was before. Sure, there may be a performance hit - but performance is not my biggest issue on these QM. 

As it stands I just whacked the rdqm shared fs space for each QM to a stupidly high number and we'll monitor. If I find that they blow up due to disk and R00* files then I can always try this param.

Not that I like the words "undocumented param" !

thanks folks,

John

John,

"give ALL the QM on a machine a large disk and let them fight out who uses it" might work in your environment (or a test environment), but it's not something I would advise as a generic approach.

A queue manager is MUCH more tolerant of a failure to allocate disk space in a queue file (where I'd consider it an APARable error if the QMgr was to end due to a disk space issue), than of a failure to write a log record (which would be expected to cause the QMgr to abort). The best generic MQ advice has long been to have a file system per QMgr for queue manager data (mainly queue files) and another file system per QMgr for the recovery logs. There should also be a further file system for /var/mqm, primarlity so that filling /var/mqm/trace or /var/mqm/errors doesn't impact QMgr operations. This sort of approach prevents file space issues in one queue manager from impacting other queue managers, but does clearly require the admin to decide how much space to allocate to each QMgr.

I wouldn't expect setting MaxReusableLogExtents to have much of a positive effect on the total amount of log space needed for linear logs (both Snnnnnnn.LOG and Rnnnnnn.LOG combined). The total amount of log spce needed for linear logs will primarily be constrained by the frequency with which you take media images. Imagine a setup where media images of the entire QMgr are taken at midnight each night. Prior to the introduction of the Rnnnnnnn.LOG files the log space would have grown throughout the day, and then when the new images were taken a bunch of log files could be deleted. New log files would then be allocated throughout the next day and that days log files could be deleted when the next images were taken. What happens with the Rnnnnnnn.LOG files is that instead of the Snnnnnnn.LOG files being deleted they are renamed as Rnnnnnnn.LOG files and then as new Snnnnnnn.LOG files are needed the Rnnnnnnn.LOG files are renamed to the appropriate Snnnnnnn.LOG files. Hence the total space needed at the end of the day should be very similar. In your case, where you let all the QMgrs fight it out then if you take the images on each QMgr at different  times then you might get some economy of scale on disk space usage, but at the expense of writing more data to disk and making it very difficult to ensure no QMgr runs out of file space for recovery logs.

Regards

Andy.

Hi Andy,

" The best generic MQ advice has long been to have a file system per QMgr for queue manager data (mainly queue files) and another file system per QMgr for the recovery logs. There should also be a further file system for /var/mqm, primarlity so that filling /var/mqm/trace or /var/mqm/errors doesn't impact QMgr operations. This sort of approach prevents file space issues in one queue manager from impacting other queue managers, but does clearly require the admin to decide how much space to allocate to each QMgr."

Is blown out of the water for RDQM. You have no choice but allocate your filesystem for all the data and logs in one fs - you can't split them. (although /var/mqm can  be seperate)

I wouldn't expect setting MaxReusableLogExtents to have much of a positive effect on the total amount of log space needed for linear logs (both Snnnnnnn.LOG and Rnnnnnn.LOG combined). The total amount of log spce needed for linear logs will primarily be constrained by the frequency with which you take media images. Imagine a setup where media images of the entire QMgr are taken at midnight each night. Prior to the introduction of the Rnnnnnnn.LOG files the log space would have grown throughout the day, and then when the new images were taken a bunch of log files could be deleted. New log files would then be allocated throughout the next day and that days log files could be deleted when the next images were taken. What happens with the Rnnnnnnn.LOG files is that instead of the Snnnnnnn.LOG files being deleted they are renamed as Rnnnnnnn.LOG files and then as new Snnnnnnn.LOG files are needed the Rnnnnnnn.LOG files are renamed to the appropriate Snnnnnnn.LOG files. Hence the total space needed at the end of the day should be very similar. In your case, where you let all the QMgrs fight it out then if you take the images on each QMgr at different  times then you might get some economy of scale on disk space usage, but at the expense of writing more data to disk and making it very difficult to ensure no QMgr runs out of file space for recovery logs

In this particular setup I use the default 60 mins for auto media imaging and transactions are reasonably short so S00 logs aren't the issue - and can be calculated reasonably well. It was the unexpected number of R000 logs that caught me out. I've even retrospectively used the formula given in the infocenter(https://www.ibm.com/docs/en/ibm-mq/9.4.x?topic=csl-how-large-should-i-make-my-log-filesystem)

LogFilesystemSize > (PrimaryFiles + SecondaryFiles + (((TimeBetweenMediaImages *2) + TimeNeededToResolveDamagedObject) * ExtentsUsedPerHour)) * LogFilePages

&

ReuseExtents <= LogDataLengthBetweenMediaImages

this number came out to no where near what I was seeing in reality in standard operation never mind a failure condition. So, forgive me if I just upped the disk space to a stupid number and watch it over time - if that means prod doesn't go down due to (cheapish) disk space but we have too much allocated - then that's a loss worth taking:-) 

John,

The calculation "LogFilesystemSize > (PrimaryFiles + SecondaryFiles + (((TimeBetweenMediaImages *2) + TimeNeededToResolveDamagedObject) * ExtentsUsedPerHour)) * LogFilePages" will give you a number of log file pages. I don't believe MQ has ever exposed a log file page size other than 4KB, and so the actual file size suggested in bytes would be 4096 times this size. If this has been a stable configuration ( I'd expect the heuristics of sizing the reuse log file pool would take time (multiple media image intervals) to react to changes) and if the size is significantly too high then I'd suggest a PMR is needed to get this investigated.

I'm afraid I can't give any insight into the rational of the file system configuration chosen for RDQM. MQ's RDQM support originated in the appliance environment at a time where MQ appliance support was restricted to circular logging. Even then the appliance team didn't follow the classic advice of seperating the QMgr data and recovery log data into different file systems, but I can't give you any insight into the origin of even that decision.

Could you choose to use NativeHA rather than RDQM ? Native HA has a big advantage over RDQM in that the queue manager understands what writes are related to persistent messaging and will transfer the minimum of data (just the recovery log data writes) between the instances. RDQM has no knowledge of what the individual writes to the file system represent and will therefore need to ship ALL of the writes to the file system between the instances. I'd expect that inhibiting the use of the Rnnnnnn.LOG files in the RDQM space would be even less attractive as RDQM would also ship all write activity associated with formatting any extent. Other than wanting each of the instances to have the same configuration I wouldn't have expected NativeHA to put any restrictions on the file system configuation for the QMgr.

Regards

Andy.

Native HA is indeed the way to go for this customer. However, they don't want to put MQ into a container so were waiting for the native HA in the non-container environment to be available. And then add a few fix packs later to pick it up :-)

Hi John,

With regards to NHA in non-container environments, we added support for running NHA and CRR on Linux (VMs bare metal etc.) in the v9.4.4 CD release, as for waiting for a few fix-packs, NHA was released way back in a 9.2 CD release.

Regards,

Martin