MQ

You may have seen a post recently by Matt Leming on how we test MQ here in Hursley Labs, however, this is not the only place where MQ for z/OS gets tested. Once a version of MQ is released, it begins being tested in the Consolidated Service Test (CST) environment. In this environment, we simulate customer-like usage with continuously running workloads with queue managers that run 24/7 alongside other IBM Z middleware products such as IMS, CICS and WAS.

Consolidated Service Test only installs a few levels of a product at a time, across our two sysplexs. The level of MQ we install is N LTS and N-1 LTS where N is the latest LTS release. From MQ 9.3.0 LTS, we also began testing N CD releases. At the time of making this blog, we have the current levels of MQ installed:

• MQ for z/OS 9.2.0 LTS
• MQ for z/OS 9.3.0 LTS
• MQ for z/OS 9.3.2 CD

We currently have a total of 23 queue managers across 3 data sharing groups, a 6-way DSG and two 4-way DSGs

Testing in CST is a little different to your traditional testing where the product is built, deployed, tested and then torn down. In CST, our Queue Managers are long-running, with our oldest queue manager existing for almost 20 years. This allows us to run long-running workloads on the queue managers to simulate customer-like usage and verify that MQ behaves as expected when thrown into a customer-like scenario. There are many different workloads that we run in the CST environment using a range of different z/OS products. 

Each month and quarter of testing produces the Recommended Service Upgrade (RSU), which sets out the levels of maintenance installed across all the products tested in CST, enabling customers following the IBM zOS Preventative Maintenance Strategy to be confident that the levels of maintenance they are installing have been tested together. 

The RSU is released every quarter, with an addendum released on the 2nd and 3rd month of that quarter. Below is a diagram of the quarterly test cycle in CST that enables us to produce the RSU:

So, what happens when a problem is identified in the CST Environment? As we run a customer-like environment, we also handle problems like a customer. This involves following the same service process as a customer would by first raising the problem in a support case with service. From there, the case is handled by Level 2 and Level 3 support, and an APAR is created to resolve the problem we have identified. The APAR and associated PTFs are then installed into the CST environment once they have closed and are tested before being included in the RSU.

The unique way the CST environment is configured allows us to test scenarios that cannot be easily performed in our test environment at the labs. CST performs disruptive tests to see how products handle worst-case scenarios and recover, for example, we intentionally fail some LPARs and IPL them in a backup location to test disaster recovery of the products. Alongside this, it enables us to run workloads that drive thousands of transactions a second over several hours, days or even weeks, with some workloads that run 24/7 across the environment.

Testing in CST does not only just validate that MQ behaves as expected, but it also ensures that MQ works alongside other IBM Products, many of our workloads will flow through several products to prove they work together in a customer-like scenario. One example is a workload that flows from IMS, through IBM MQ into CICS and then back again. An additional benefit from maintaining and testing these products in such an integrated manner in CST results in the development, test and support teams from all these products having to collaborate and work together.

Although extensive testing is performed here in Hursley Labs, it does not eliminate the possibility for problems to make their way out into the wild. CST is the last line of defence for testing MQ and finding problems before customers experience them. 

You can find out more about Consolidated Service Test from our webpage