Circular Logging Vs Linear logging with automatic management
Anthony_Beardsmore
Published on 09/05/2019
This blog post is primarily a response to a question raised at the IBM Technical Conference last week. It seemed like the full answer would be of interest to a wider audience hence this follow up.
With MQ 9.0.2 (now 9.1 LTS) we introduced a number of improvements around the management of linear logs in MQ. These are well covered in depth by this blog post by Mark Whitlock from that time.
For full details, I’d encourage you to review that post and the knowledge center documentation, but in a nutshell you can now:
- Automate media imaging (and have it occur at the ‘best‘ time from a queue manager performance point of view)
- Automate the reuse of logs which are no longer required for recovery (because they have been made redundant by a newer media image).
- Get substantially Improved performance (because log reuse is faster than creation and formatting of new log extents.)
- With those improvements in mind, the question we were asked was ‘why would I ever use circular logging’ and as a follow on ‘why doesn’t IBM merge these two options and/or make linear managed logging the default’.
Two aspects of the system are of interest in answering this question:
- Performance – total data throughput requirements of your applications.
- Storage usage – how much storage is available to the queue manager (or to put it another way – the total cost to you of increasing this storage allocation).
With circular logging, storage usage is kept to an absolute minimum since as soon as all transactions touched in a given log file have been committed, that log is available for reuse. This means that on a well configured system, new (secondary) log files are very rarely required, and primary log files may be rotated through many times a second, keeping all disk usage in a small and tightly defined space. This can have further benefits, for example if all disk access can stay within the cache of your storage system for even greater throughput.
With linear logging – even with all of the new options available – performance and storage usage will still be gated on the frequency of media imaging. Log extents cannot be reused until all updates to queues held in that file have been committed to a more recent media recovery image – and even with low queue depths and high frequency of media imaging, this is likely to be in the order of minutes or hours. On a system which regularly holds deep queues (which are costly to snapshot) these snapshots are likely to be fewer and further between – meaning significantly more log space is required.
Because of these differences, there will still be significant real world advantages and disadvantages to the two approaches, and it comes down to the expected load on the system and performance and storage requirements. For this reason we are not likely to remove the option to use circular logging or modify the default for queue managers – at least any time soon.
by Anthony_Beardsmore
2 comments on"Circular Logging Vs Linear logging with automatic management"
- RayMay 09, 2019
Excellent write-up.
- PPotkayMay 09, 2019
Thank you Anthony for writing up the follow up and making it publicly available.
-Peter