Introduction

Native High Availability (HA) Cross Region Replication (CRR) was introduced in IBM MQ 9.4.2 (announcement letter) to securely replicate a queue manager asynchronously across regions. The team have created the following video that provides a high-level overview of its capability: https://www.youtube.com/watch?v=xvukMdiYb-0. This blog will guide you through the process of deploying two IBM MQ Native HA queue managers across two Red Hat OpenShift cluster namespaces; one as Live and one as a Recovery group.

Deploying and configuring queue managers on Kubernetes can be achieved using YAML resources directly, and the product documentation details the steps required to build-up the relevant YAML resources. IBM MQ provides a helm chart, with various samples for different configurations, to generate and install appropriate YAML files which deploy IBM MQ into your chosen Kubernetes environment. Red Hat OpenShift is one example of a Kubernetes environment. In this blog, we will be using the OpenShiftNativeHACRR sample. This sample uses the OpenShift client command ('oc') which is an built upon the standard Kubernetes command ('kubectl').

The helm chart and samples can be found in the following GitHub repo: https://github.com/ibm-messaging/mq-helm

Following deployment of the sample, you will have configured Native HA across two clusters securely communicating with each other, and they will be asynchronously performing cross region replication, as shown here:

Deploying the Recovery group

With the crr-a and crr-b projects created in the two clusters, log in to Cluster B with ‘oc login’ and issue the following command from the samples' deploy directory, to install the Native HA CRR Recovery group:

./install.sh qm-recovery crr-b

The install.sh script will create the sample self-signed certificate secrets and deploy the Native HA CRR group using the ‘helm install’ command. It will pull the latest IBM MQ 9.4.2 container image from the public container repository: icr.io/ibm-messaging/mq.

Validate the deployment

To validate that the deployment has been successful, you can either:

1. Issue ‘oc get pods’ to view the three deployed pods, and the Recovery/Leader pod will be the one showing 1/1 in the READY column:
   
   

2. Log into the Cluster’s web console, view: Workloads -> Pods (ensuring that you have the ‘crr-b’ project selected), and you will see three Running pods with the Recovery/Leader pod identifying itself with 1/1 in the Ready column:
   (for the remainder of this blog, we will check and run actions from the command line)

The next validation check would be to ask IBM MQ, via the dspmq command, what state it believes the Native HA CRR group configuration is in, using the new -g option. To view the groups' state, as seen by the Recovery/Leader, we can issue the following command on that pod:

oc rsh secureapphelm-ibm-mq-0 dspmq -o nativeha -g

The output of this command will show that this pod is the Leader of the Recovery group and currently ‘Waiting for connection’ from the Live group: [It is expected that the QUORUM value is 1/3]

QMNAME(secureapphelm)                                     ROLE(Leader) INSTANCE(secureapphelm-ibm-mq-0) INSYNC(yes) QUORUM(1/3) GRPLSN() GRPNAME(beta) GRPROLE(Recovery)
 GRPNAME(beta) GRPROLE(Recovery) GRPADDR(Unknown) GRPVER(9.4.2.0) GRSTATUS(Waiting for connection) RCOVLSN() RCOVTIME() BACKLOG(Unknown) INSYNC(Unknown) SYNCTIME() ALTDATE(2025-03-06) ALTTIME(09.43.07)
 GRPNAME(alpha) GRPROLE(Unknown) GRPADDR(Unknown) GRPVER(?.?.?.?) CONNGRP(no) GRSTATUS(Unknown) RCOVLSN() RCOVTIME() ALTDATE() ALTTIME()

Success!

Deploying the Live group

Whilst still connected to the Recovery group’s project, obtain its route address so it can be used by helm to configure the Live group and be able to securely replicate data once installed:

export HACRR_ROUTE=$(kubectl get route secureapphelm-ibm-mq-nhacrr -n crr-b -o jsonpath="{.spec.host}")

echo $HACRR_ROUTE\(443\)

This will output the route address similar to:

secureapphelm-ibm-mq-nhacrr-crr-b.apps.opal.cp.fyre.ibm.com(443)

Edit secureapp_nativeha.yaml to set this as the address value, within the section describing the Recovery group's deployment information:

Then, issue the following command from the samples' deploy directory, to install the Native HA CRR Live group:

./install.sh qm-live crr-a

Validate the deployment

After validating the deployment of the Live CRR group by ensuring that all of the pods are Running and one has 1/1 in the Ready column, issue the same dspmq command on that Live/Active pod and again on the Recovery/Leader pod. What you will see is:

Live/Active pod:

QMNAME(secureapphelm)                                     ROLE(Active) INSTANCE(secureapphelm-ibm-mq-0) INSYNC(yes) QUORUM(3/3) GRPLSN(<0:0:12:47185>) GRPNAME(alpha) GRPROLE(Live)
 GRPNAME(alpha) GRPROLE(Live) GRPADDR(Unknown) GRPVER(9.4.2.0) GRSTATUS(Normal) RCOVLSN(<0:0:12:47185>) RCOVTIME(2025-03-06T10:25:41.014673Z) INITLSN(<0:0:9:7096>) INITTIME(2025-03-06T10:24:50.098481Z) LIVETIME(2025-03-06T10:24:55.113744Z) ALTDATE(2025-03-06) ALTTIME(10.25.41)
 GRPNAME(beta) GRPROLE(Recovery) GRPADDR(secureapphelm-ibm-mq-nhacrr-crr-b.apps.opal.cp.fyre.ibm.com) GRPVER(9.4.2.0) CONNGRP(yes) GRSTATUS(Normal) RCOVLSN(<0:0:12:47185>) RCOVTIME(2025-03-06T10:25:41.014673Z) BACKLOG(0) INSYNC(yes) SYNCTIME(2025-03-06T10:29:45.338469Z) ALTDATE(2025-03-06) ALTTIME(10.29.37)

Recovery/Leader pod:

QMNAME(secureapphelm)                                     ROLE(Leader) INSTANCE(secureapphelm-ibm-mq-0) INSYNC(yes) QUORUM(3/3) GRPLSN(<0:0:12:47185>) GRPNAME(beta) GRPROLE(Recovery)
 GRPNAME(beta) GRPROLE(Recovery) GRPADDR(secureapphelm-ibm-mq-nhacrr-crr-b.apps.opal.cp.fyre.ibm.com) GRPVER(9.4.2.0) GRSTATUS(Normal) RCOVLSN(<0:0:12:47185>) RCOVTIME(2025-03-06T10:25:41.014673Z) BACKLOG(0) INSYNC(yes) SYNCTIME(2025-03-06T10:30:26.346861Z) ALTDATE(2025-03-06) ALTTIME(10.25.53)
 GRPNAME(alpha) GRPROLE(Live) GRPADDR(Unknown) GRPVER(9.4.2.0) CONNGRP(yes) GRSTATUS(Normal) RCOVLSN(<0:0:12:47185>) RCOVTIME(2025-03-06T10:25:41.014673Z) INITLSN(<0:0:9:7096>) INITTIME(2025-03-06T10:24:50.098481Z) LIVETIME(2025-03-06T10:24:55.113744Z) ALTDATE(2025-03-06) ALTTIME(10.30.27)

The key information you can see in these outputs are that both groups:

• Are in-sync:
  • INSYNC(yes)
• Have quorum within their own groups: 
  • QUORUM(3/3)
• Know their respective names and roles:
  • GRPNAME(alpha) GRPROLE(Live), and
  • GRPNAME(beta) GRPROLE(Recovery)

Success!

Conclusion

You have now deployed an IBM MQ Native HA CRR configuration across two cluster namespaces, using helm. Additional details on the roles, example state transitions when performing a planned switchover, and rebasing behaviour can be found in the IBM MQ documentation: https://www.ibm.com/docs/en/ibm-mq/9.4?topic=containers-native-ha-cross-region-replication

It will be great to hear from you about your deployments and how you are using Native HA CRR within your organisation.