MQ

 View Only

MQ for z/OS: Page set performance and best practice

By Anthony Sharkey posted Tue February 08, 2022 09:46 AM

  

 

It was recently brought to my attention that there is not a lot of performance information relating to MQ for z/OS page sets, so this blog was created to try to address that deficit.

 

Firstly let’s remind ourselves of the relationship between messages, buffers, buffer pools and page sets. A buffer pool is associated with one or more page sets; each page set is associated with a single buffer pool as demonstrated in figure 1.

 

Figure 1: Buffers, buffer pools and page sets.

Buffers, buffer pool and page set

When an MQ message is put on to a non-shared queue, the queue manager stores the data on a page set in such a way that it can be retrieved when a subsequent operation gets a message from the same queue. If the message is removed from the queue, space in the page set that holds the data is later freed for reuse. As the number of messages held on a queue increases, so the amount of space used in the page set increases, and as the number of messages on a queue reduces, the space used in the page set reduces.

To reduce the performance cost of writing data to and reading data from the page sets, the queue manager buffers the updates into processor storage. The amount of storage used to buffer the page set access is controlled through IBM MQ objects called buffer pools.

For efficiency, IBM MQ uses a form of caching whereby messages (and object definitions) are stored temporarily in buffers before being stored in page sets on DASD. Short-lived messages, that is, messages that are retrieved from a queue shortly after they are received, might only ever be stored in the buffers. This caching activity is controlled by a buffer manager, which is a component of IBM MQ.

What is a page set?

The IBM MQ Technical overview has the following definition of a page set:

 

“A page set is a VSAM linear data set that has been specially formatted to be used by IBM MQ for z/OS. Page sets are used to store most messages and object definitions.

 

The exceptions to this are global definitions, which are stored in a shared repository on Db2®, and the messages on shared queues. These are not stored on queue manager page sets.

 

IBM MQ page sets can be up to 64GB in size, and each page set is identified by a page set indicator (PSID), an integer in the range 00 through 99.”

How many page sets do I need?

The IBM MQ documentation suggests there at least five page sets defined:

  • A page set reserved for object definitions (page set zero)
  • A page set for system-related messages
  • A page set for performance-critical long-lived messages
  • A page set for performance-critical short-lived messages
  • A page set for all other messages.

 

If you are a reliant on MQ clustering, you might also want to put SYSTEM.CLUSTER.* queues in their own buffer pool and page set.

 

Additional page sets can be added dependent on application needs. For example, if you are going to have several queues which will need to store a very large number of messages, it might be beneficial to give each of those queues their own page set.

 

How large should the page sets be?

The IBM MQ documentation section “Planning your page sets and buffer pools” describes what factors might affect the size of your page sets.

 

Note that if a page set needs to expand, the performance of the MQPUT(s) driving that expansion may be affected until the expansion is complete. How long that delay is, will depend on the size of the expansion to be completed.


Configuration of page sets 

Placement of page sets

In our environment, we typically put each page set on its own volume. This is not necessary but if you plan to place multiple page sets on the same volume, whether from the same queue manager or different queue managers, it is worth reviewing the RMF DASD reports for those volumes to ensure there is not contention occurring.

 

Should MQ page sets be striped?

The benefits of striped MQ page sets are less obvious than for MQ log data sets due to the nature of the I/O patterns. These patterns are discussed further in the section “When are page sets used?

 

In particular, page set read is single page per I/O and this gets no benefit from striped data sets.

 

Does High Performance FICON® (zHPF) help performance?

High Performance FICON (zHPF) is a channel I/O architecture that is designed to improve the execution of small block I/O requests.

 

Generally, zHPF does reduce I/O response times, but it is worth confirming in your own environment.

 

In our measurements, enabling zHPF resulted in a reduction in elapsed times for both single page I/O and multiple page writes of approximately 30%.

 

Will encrypted page sets affect performance?

Support for encrypted page sets, active logs and shared message data sets (SMDS) was implemented in MQ for z/OS in version 9.2.

 

The impact of data set encryption on MQ page sets is minimal, but realistically data set encryption would be applied across all of the MQ data sets, including active logs, archive logs and SMDS, and there is a larger impact when applying data set encryption to MQ logs.

 

The performance impact of encrypting your MQ queue manager is discussed in detail in the up MQ for z/OS v9.2 performance report, which can be found on the MQ Performance documents repository.

When are page sets used?

As previously discussed in this blog, MQ will initially store non-shared messages in buffer pools.

 

For best MQ API response times, page set usage should be kept to a minimum. Where this is not possible, perhaps due to some application outage causing deep queues or messages remaining on MQ queues for an extended period of time, the buffer pools should be sized to try to avoid synchronous I/O.

 

Writing to page set

Messages stored in buffers are written to page set under a limited set of criteria:

  • When they have been in the buffer pool for 2 log checkpoints. At this point MQ’s deferred write processor/task (DWP or sometimes DWT) will write the messages to page set asynchronously.
  • When the buffer pool usage exceeds the deferred write threshold, typically at 85% full. Once more, MQ’s DWP will write the messages to page set asynchronously.
  • When the buffer pool usage exceeds the buffer pool threshold, typically at 95% full. In this case, the DWP has not been able to keep pace with the rate of messages arriving on the buffers, so subsequent writes are performed synchronously, which will affect the performance of the application putting the messages to the MQ queues.

 

It is important to note that the deferred write processor runs asynchronously to those applications performing MQ APIs. The DWP is able to batch up I/O requests, writing up to 16 pages per I/O request, i.e. writing 64KB of data.

 

When the buffer pool usage reaches the 95% full threshold, I/O to page set is performed synchronously, with 1 page per I/O request.

 

With more memory generally being available on IBM z, as well as MQ for z/OS supporting 64-bit buffer pools, it is possible to configure your queue manager to minimize synchronous page set writes.

The use of oversized buffer pools, i.e. where the buffer pool is sized at 105% of the size of the page set, means that synchronous writes can be avoided. This is a simple way to ensure application performance is not affected by I/O to page set when the buffer pool reaches capacity. Oversized buffer pools do require sufficient buffers being allocated to the buffer pool.

It is simpler to determine oversized buffer pool sizes when there is a 1:1 relationship between buffer pools and page sets.

Remember that if page set expansion occurs, the buffer pool may also need to be re-sized.

 

Reading from page set

Once the messages are stored on page set and no longer available in buffers, they must be read back from page set before they can be gotten by applications.

 

Read page set I/O is performed 1 page per request, which will be primarily affected by the DASD response time.

 

Later on in this blog, we will show the impact of having to rely on page set I/O for MQGET and compare that with the performance where the messages are retained in the buffer pools.

What can I use to monitor my MQ page sets?

There are a number of options to monitor your MQ page set usage.

 

The RMF DASD report can display information relating to the volume or storage group.

In our configuration we have created separate storage groups for each type of page set, i.e. page set ‘00’ uses storage group ‘MQPAGE0’ which are then assigned a set of volumes. Similarly, page set ‘01’ uses storage group ‘MQPAGE1’, and so on.

 

The RMF DASD report for a workload where the queue manager is performing large amounts of write to and read from page set 01 shows the following:

RMF Direct Access Device Activity report

 

The z/OS RMF Report Analysis section “Pending time overflow” discusses the attributes in this report. As a guide, if you see non-zero values in the delay-type or IOSQ time fields, it may be worth engaging with your storage administrator to determine if there is any action that can be taken.

 

If there are multiple queue managers on the same LPAR that share the same storage group for their page sets, this RMF DASD report may not offer sufficient granularity to be able to determine which queue manager is affected by any I/O delays. Under those circumstances it may be preferable to configure separate volumes for each queue managers individual page sets.

  

RMF record type 42 – DFSMS statistics and configuration records, in particular subtype 6 (or RMF 42(6), record DASD data set level I/O statistics. These records can provide data on the particular MQ page data set, including information such as the number of I/O’s, response time, whether the data was read from disk cache. Unfortunately RMF does not provide a formatted report for this data.

  

The MQ statistics trace, specifically class ‘01’ generates both page set and buffer pool statistics. You can use the MQSMF program as part of MQ SupportPac MP1B’s to format the page set and buffer pool data. An example of the page set report is provided below:

Page set report

 

In this example, all of the pages written to page set 01 were written outside of checkpoint – i.e. the deferred write threshold of the buffer pool had been reached.

 

The page set report also shows each write contained 16 pages per I/O with an average time of 1.89 milliseconds.

 

It is useful to take into context the buffer pool statistics data from the same SMF interval:

Buffer pool report

 

The buffer pool report shows that there were 23115 WIO (write) requests and 0 IMW (immediate / synchronous writes) requests.

 

In this particular case, the buffer pool only contains 20,000 pages, which is too small for this particular workload – and as a result the workload is performing I/O to and from the page set which would be avoided with an appropriately sized buffer pool.

Considerations for backup and recovery

The size of the page set determines the time taken to recover a page set when restoring from a backup, as a large page set takes longer to restore.

 

Recovery of a page set also depends on the time the queue manager takes to process the log records written since the backup was taken; this time period is determined by the backup frequency.

 

To calculate the backup frequency, start by determining the target total recovery time. This will consist of:

  • The time taken to react to the problem.
  • The time taken to restore the page set backup copy. For example, we can restore approximately 6200 cylinders of 3390 data per minute from and to DS8800 DASD using DFDSS.
  • The time the queue manager requires to restart, including the additional time needed to recover the page set. This depends most significantly on the amount of log data that must be read from active and archive logs since that page set was last backed up.

For further information on backup performance, please refer to MP16’s “How often should a page set be backed up” section.

The impact of page set performance on workload

To see the impact of page set I/O on workload we will look at two workloads in a range of configurations.

 

The first workload demonstrates the impact of page set read I/O on the transaction round-trip time. In addition, the need to perform page set I/O add to the transaction costs, primarily in the MQ MSTR address space.

 

The second workload streams persistent messages from one queue manager to a second queue manager on a separate z/OS LPAR. The workload allows a build-up of messages on the transmit queue before starting the transfer, which ensures the channel is able to send full batches of 200 messages. By allowing the build-up of messages, the workload relies primarily on the deferred write task to write messages to page set, as well as requiring a high proportion of the page set reads by the MQGETs requested by the MCA channel.

 

It should be noted that these measurements are micro-benchmarks that are very much oriented to MQ and do not contain non-essential processing. As such, the impact of making similar changes to the MQ configuration in a more realistic environment may result in a smaller improvement.

 

Workload 1

The first workload pre-populates a single MQ queue with a set of messages, the number depending on the size of the messages:

  • 500,000 non-persistent messages of 1KB.
  • 30,000 non-persistent messages of 64KB.

 

Once the queues have been loaded with the messages, a single application will randomly select a message to MQGET and then subsequently MQPUT back to the queue. The application repeats this process for multiple SMF intervals.

 

Each set of message size workloads is run in 3 configurations:

  1. Buffer pool of 20,000 buffers
  2. Buffer pool of 100,000 buffers
  3. Buffer pool of 1,048,576 buffers

 

1KB workload

 

Buffers

20,000

100,000

1,048,576

Transaction Rate / second

 

4,655

6,525

50,490

Round-trip time (microseconds)

214

153

20

Transaction cost (microseconds)

(cost: MQ MSTR and application)

40

27

20

Wait time

(round-trip minus cost)

164

126

0

% of MQGETs requiring page set I/O

100

77

0

Elapsed read page time (microseconds)

163

169

N/A

Weighted page set read time

 

Based on page read time multiplied by percentage of MQGETs requiring page set I/O

 

 

163

 

 

130

 

 

N/A

 

Notes:

  • A 1KB message will use a single 4KB page – so each message that is read from page set will be resolved by a single page read.
  • The time spent performing page set read I/O accounts for most, if not all, of the wait time, i.e. the difference between round-trip time and transaction cost.
  • Due to the nature of the workload, there was no impact to the round-trip time from the DWP writing messages to page set, and there are no immediate writes to page set.
  • The configuration with sufficient buffers such that page set reads are avoided makes a significant difference to both transaction rate (7-10 times) and cost.

 

 

64KB workload

 

Buffers

20,000

100,000

1,048,576

Transaction Rate / second

 

351

478

22,109

Round-trip time (microseconds)

2842

2089

44

Transaction cost (microseconds)

(cost: MQ MSTR and application)

173

122

48

Wait time

(round-trip minus cost)

2669

1969

-4

% of MQGETs requiring page set I/O

100

73

0

Elapsed read page time (microseconds)

163

164

N/A

Elapsed read message time (microseconds)

17 pages per message

2771

2788

N/A

Weighted read message time

Based on read message time multiplied by percentage of MQGETs requiring page set I/O

 

2771

 

2035

 

N/A

 

Notes:

  • A 64KB message will use 17 pages to store the message and MQ implementation headers – so each message that is read from page set will require 17 I/O requests, i.e. the larger the message needing to be read from page set, the greater the impact of DASD response time to read each page.
  • Page set read time accounts for the majority of wait time.
  • In some cases, “read message time” plus “transaction cost” does exceed round-trip time, which is due to some asynchronous work taking place – for example the DWP writing messages to page set.
  • Page set reads for 64KB messages were taking 2.7 milliseconds – a time which is eliminated entirely when the buffer pool is sufficiently large to contain the workload. By removing page set read time for this workload, we can improve throughput by over 40 times.

 

Enabling zHPF in these measurements reduced our single page read performance from 160-170 microseconds to 118 microseconds, a reduction of 28%. Similarly the time taken to write the 16 pages/IO as part of DWP (not reported) decreased 31%.


 

Workload 2

The second workload uses a single batch application to put approximately 12GB of persistent messages to a remote queue. Once there is 2GB of message data on the queue, the MQ channel is started and messages are sent to the remote queue manager to be consumed.

 

For the purposes of this blog, we are only interested in the performance of the sending queue manager.

 

As the workload is streamed, round-trip time is not an appropriate metric, and instead we will report the rate at which the messages are processed end-to-end.

 

For each configuration, the workload will be run with:

  • 10KB persistent messages
  • 1MB persistent messages

 

Each set of message size workloads is run in 4 configurations:

  1. Buffer pool of 200,000 buffers
  2. Buffer pool of 200,000 buffers with zHPF enabled
  3. Buffer pool of 1,048,576 buffers

 

Despite these workloads using persistent messages and transporting data over a 10Gb network, the constraining factor is typically the elapsed time for the MQGET by the MCA channel.

 

10KB persistent workload

 

Buffers

200,000

200,000

1,048,576

zHPF

No

Yes

No

Readahead

No

No

No

Transaction Rate / second

 

1,954

2,609

13,497

Transaction cost

(cost: MQ MSTR, MQ CHIN, application and network)

108

113

86

DWP I/O time (16 pages)

2,350

1,440

1,800

IMW I/O time (1 page)

217

173

375

Elapsed time to read page

175

128

N/A

Elapsed time to read message

3 pages per message

525

384

0

Percentage of messages requiring page set read

87%

87%

0

 

Notes:

  • Measurements and elapsed times are CPU microseconds, unless otherwise stated.
  • Enabling zHPF resulted in:
    • DWP I/O times decreasing 38%,
    • Immediate (synchronous) writes decreasing 20%,
    • Page set reads elapsed times decreasing 26%,
    • Overall throughput increase of 33%.
  • Providing sufficient buffers that the messages could be gotten by the MQ CHIN without requiring page set reads, once more gives the best performance at the lowest cost, and the throughput begins to be constrained by network latency and/or MQ log limits.

 

 

1MB persistent workload

 

Buffers

200,000

200,000

1,048,576

zHPF

No

Yes

No

Readahead

No

No

No

Transaction Rate / second

 

22

29

137

Transaction cost

(cost: MQ MSTR, MQ CHIN, application and network)

6,630

6,620

5,060

DWP I/O time (16 pages)

2,350

1,590

2,360

IMW I/O time (1 page)

250

227

N/A

Elapsed time to read page

175

126

N/A

Elapsed time to read message

257 pages per message

(CPU milliseconds)

44.9

32.2

0

Percentage of messages requiring page set read

95%

95%

0

 

Notes:

  • Measurements and elapsed times are CPU microseconds, unless otherwise stated.
  • Enabling zHPF resulted in:
    • DWP I/O times decreasing 32%,
    • Immediate (synchronous) writes decreasing 9% - although very few immediate writes required,
    • Page set reads elapsed times decreasing 28%,
    • Overall throughput increase of 32%.
  • Providing sufficient buffers that the messages could be gotten by the MQ CHIN without requiring page set reads once more gives the best performance at the lowest cost, and the throughput begins to be constrained by network latency and/or MQ log limits.

 

 

Summary

Best performance on non-shared queues will come when the messages can be processed entirely in buffer pools, but this is not always possible.

 

When the workload may require in page set I/O, ensure:

  • The page sets are allocated on the most responsive DASD.
  • zHPF is enabled to reduce DASD response times.
  • There is no contention at channel path, volume, or data set level.

 

It is also worth considering whether oversized buffer pools might reduce the impact of synchronous writes to page sets on your applications.

 

From a general performance tuning perspective, it is simpler to determine the appropriate sizes of the buffer pools when there is a 1:1 relationship between buffer pools and page sets – and this is true regardless of whether using oversized buffer pools.

 

 

Finally, if the performance of applications using non-shared queues is of absolute importance, it is worth reminding you of the buffer pool attribute “PAGECLAS”, which is an optional parameter describing the type of virtual storage pages used for backing the buffers in the buffer pool.

When using PAGECLASS(FIXED4KB) it is imperative that there is sufficient real storage available on the LPAR. The blog “IBM MQ Little Gem #51: PAGECLAS” pulls together a lot of information about this attribute.

 

 



#IBMMQ
#Integration
#MQ
#z/OS
0 comments
145 views

Permalink