MQ

Introduction

Slow system performance can be time consuming and difficult to troubleshoot.  Applications built to communicate using IBM MQ have some advantages over applications using other communication styles when performance problems occur. MQ's Queues act as a buffer at peak workloads, safely caching requests until the system can catch up, and they allow improved visibility into a problem, with increasing queue depths allowing fast identification of bottlenecks. Added to this, MQ offers a mature and comprehensive set of statistics and monitoring which allow detailed analysis of problems.

However, despite these advantages it can be hard work to pull together all the information needed to find the root cause of a slowdown, so to make troubleshooting simpler we are introducing a new “performance check” tool into IBM MQ V9.4.2 CD and V9.4.0.10 LTS releases.  The tool is intended to help MQ system administrators to understand their MQ workload, and to help application developers tune their applications' usage of MQ.

The performance tool is a single command - mqperfck - which produces an HTML report containing a summary of key information about queue and channel workloads. It is designed to have a minimal impact on the system. The output is mostly based on statistics that are already collected by the queue manager like resource statistics, and queue/channel status.

How to Use the Report

The performance tool is not a replacement for existing monitoring and alerting tools. Instead, it provides a snapshot of a system activity.  We recommend that system administrators run the tool for a minute or two when the system is busy, and again when it is quiet to get a comparison of different workloads.  Later, if problems occur, then they should run the tool again and compare the reports to see what’s changed.

Application developers can use the report iteratively during the development lifecycle to check that their application is making efficient use of MQ. For instance, the report can help with tuning the balance of producers and consumers, and ensuring the application is making the best use of the MQI.

Running the report

When you run mqperfck you can specify a queue manager, a list of queues and sender channels to be included in the report, and the number of data collection iterations that should be run. 

For example, the following command runs the report for two minutes (12 iterations of approximately 10 seconds), for two queues (TESTQ and TESTQ2), and a single sender channel (TO.RECEIVER.QM).  The sender channel’s transmission queue will be included in the report automatically.  The report will be written in the /var/mqm/errors directory.

An example of the report’s output is included below with comments on some of the key statistics and how you might use them.

Example Report

The performance tool's report has the following sections: 

Section 1:  A header section, comprising information about the system and MQ version.

Section 2: System and queue manager statistics

The screenshot below shows both the header section and system CPU statistics subsection. The same layout is used by all the statistics sections of the report: On the left is the statistic name; the middle columns show the minimum, maximum and average values seen for the duration of the report; and on the right is a list of the raw values seen for the statistic in each interval / iteration.  The example report ran for three intervals so three raw values have been recorded for each statistic.

The queue manager statistic section includes subsections on queue manager CPU usage and the queue manager's transaction log (see below). 

Good logger performance is critical to the queue manager - particularly for persistent workloads.  A key metric here is “Log – write latency”.  You should monitor this value and, if the system is performing poorly, compare this to values when the system was working well.  Slow log writes - for example taking multiple milliseconds - are likely to have an impact on overall queue manager performance.

Section 3:  Queue statistics for each queue named in the command parameters

The queue statistics sections have a wealth of information about activity on each queue. The screenshot below shows the subsection for MQPUT statistics (how many messages are being put to the queue) and includes information about queue lock contention, and about queue avoidance^*. 

(* See section 5.6.1.1 of https://ibm-messaging.github.io/mqperf/MQ_Performance_Best_Practices_v1.0.1.pdf for more information on queue avoidance).

There are a similar subsections for General queue stats (number of applications with the queue open for input and output, message expiry etc.), MQGET stats (including number of rollbacks, and failing gets), and MQOPEN, MQCLOSE and Extended stats (shown below).  Use the queue section to understand the workload going through the queue, and how efficiently applications are using the queue – for instance, are they repeatedly opening and closing the queue when it could be kept open? Are messages being rolled back repeatedly? Are applications making inefficient use of message matching or selectors etc.

Section 4: Channel status for each channel

As shown below, mqperfck has an option to report on channel status (only sender channels in the initial release). This is essentially the same information you'd get from the MQCMD_INQUIRE_CHANNEL_STATUS PCF command, so you can see how much data is flowing over the channel and get an idea of the network latency, and time spent compressing data and running user exits.

5. Queue Summary / Analysis

The final section in the report is a summary of activity on each queue. 

Conclusion 

The mqperfck command pulls together various sources of information into an easily readable report.  You can use this report to help understand activity across the queue manager and selected queues and channels. Run the report under light and heavy workloads to troubleshoot performance problems, and to help to tune applications to make efficient use of MQ. 

For more information about the fields in the report, see the interpretation guide at: https://ibm.biz/mqperfck