IBM MQ offers a number of solutions to run a queue manager in High Availability(HA) configuration. The solutions offered by MQ for HA are Queue manager clusters, HA clusters, Multi-instance queue managers. HA RDQM is a part of multi-instance setup which allows instances of the same queue manager configured on each node in a group of three Linux servers. One of the three instances is the active instance. Data from the active queue manager is synchronously replicated to the other two instances, so one of these instances can take over in the event of some failure. The grouping of the servers is controlled by Pacemaker, and the replication by DRBD.
Summary: This blog explains detailed procedure of migrating regular IBM MQ Queue Manager to RDQM HA Queue manager.
The procedure involves the following steps to migrating an existing queue manager to become a high availability (HA) replicated data queue manager (RDQM) by backing up its persistent data, then restoring the data to a newly created RDQM queue manager that has the same name.
1. Back up the existing queue manager:
- Stop the existing queue manager by issuing a wait shutdown command endmqm -w, or an immediate shutdown command endmqm -i. This step is important to ensure the data in the backup is consistent.
- Determine the location of the queue manager data directory by viewing the IBM MQ configuration file, mqs.ini. On Linux, this file is located in the /var/mqm directory. Locate the Queue Manager stanza for the queue manager in the file. If the stanza contains a key named DataPath then its value is the queue manager data directory. If the key does not exist, then the queue manager data directory can be determined using the values of the Prefix(/var/mqm) and Directory keys. The queue manager data directory is a concatenation of these values, of the form prefix/qmgrs/directory. For example if we have a queue manager QM1 then data path will be '/var/mqm/qmgrs/QM1'. Create a backup of the queue manager data directory. For example, to back up the data directory for a queue manager you can use the following command. tar -cvf QM1-data.tar.gz -C queue_manager_data_dir
- Determine the location of the queue manager log directory by viewing the IBM MQ queue manager configuration file qm.ini. This file is located in the queue manager data directory. The queue manager log directory is defined as the value of the Log Path key in the Log stanza. For information out the stanza, see Log stanza of the qm.ini file. Create a backup of the queue manager log directory. For example, to back up the log directory for a queue manager you can use the following command.
tar -cvf QM1-log.tar.gz -C queue_manager_log_dir .
- Create a backup of any certificate repositories used by the queue manager if they are not located in the queue manager data directory. Ensure that both the key database file and the password stash file are backed up.
- The existing queue manager is no longer required, so it can be deleted.
2. Environment Setup for MQ RDQM setup:
Follow the below mentioned Requirements steps for RDQM HA setup:
a. System requirements You must need 3 servers that are to be part of the HA group.
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=availability-requirements-rdqm-ha-solution
b. Each node requires a volume group named drbdpool
i. First issue pvscan command to check the whilch physhical volume has space
[root@LinuxRHEL11 ~]# pvscan
PV /dev/vda4 VG drbdpool lvm2 [<50.11 GiB / <22.98 GiB free]
PV /dev/vda2 VG rhel lvm2 [248.88 GiB / 0 free]
Total: 2 [298.99 GiB] / in use: 2 [298.99 GiB] / in no VG: 0 [0 ]
ii. Then go and create drbd volument group
vgcreate drbdpool /dev/vda4
iii. Issue vgs command to see the volume groups
[root@LinuxRHEL11 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
drbdpool 1 7 0 wz--n- <50.11g <22.98g
rhel 1 2 0 wz--n- 248.88g 0
c. Setting up passwordless SSH :
You can set up passwordless SSH so that you only need issue configuration commands on one node in the HA group.
Follow the steps mentioned in :https://www.ibm.com/docs/en/ibm-mq/9.2?topic=solution-setting-up-passwordless-ssh
d. On each node, as root, complete the following steps to set up sudo access for the mqm user Edit the sudoers file by using the visudo command:
visudo
Search for the line "## Allows people in group wheel to run all commands" and add the following text below the line:
#%mqm ALL=(ALL) ALL
Search for the line "## Same thing without a password" and add the following text below the line:
%mqm ALL=(ALL) NOPASSWD: ALL
e. Go and install the MQ 9.2 RDQM packages (present under MQ 9.2 Serverimage/Advanced/RDQM) , Follow the instructions to install the MQ RDQM https://www.ibm.com/docs/en/ibm-mq/9.2?topic=multiplatforms-installing-rdqm-replicated-data-queue-managers
- First route inside MQ 9.2 Serverimage/Advanced/RDQM/PreReqs directory and route to el7 / el8 depending on Linux RHEL 7 / 8 - if RHEL 7 -el7 , if RHEL 8 - el8
- Route to Advanced/RDQM/PreReqs/el8/kmod-drbd-9/ and check for modver by running ./modver - it will return the supported kmod-drbd package needs to be installed on the machine.
Ex: Install kmod-drbd by running yum install kmod-drbd-9.0.23_4.18.0_193-1.x86_64.rpm
- Install the required DRBD utilities, for RHEL 8.x you run the following command: Ex: yum install Advanced/RDQM/PreReqs/el8/drbd-utils-9/*
- Install Pacemaker. For example, for RHEL 8.x you run the following command: Ex: yum install Advanced/RDQM/PreReqs/el8/pacemaker-2/*
- If MQ is already installed, go and install RDQM yum install Advanced/RDQM/MQSeriesRDQM*
If MQ is not installed , then accept the license and install the RDQM dependent components: yum install MQSeriesGSKit* MQSeriesServer* MQSeriesRuntime*
f. Add mqm user to HAClient group : the haclient group is created during installation of Pacemaker, add mqm user to haclient group on all 3 server (Nodes)
useradd -G haclient mqm
Issue id mqm to confirm , mqm is part of haclient
[root@LinuxRHEL11 ~]# id mqm
uid=998(mqm) gid=1001(mqm) groups=1001(mqm),189(haclient)
3. Prepare the primary RDQM node: a. Create a new RDQM queue manager with the same name as the queue manager that you backed up. Ensure the filesystem allocated for the RDQM queue manager by crtmqm is big enough to contain the data, primary logs and secondary logs for the existing queue manager, plus some additional space for future expansion.
For example: crtmqm -sx QM1
b. Determine the primary RDQM node for the queue manager using rdqmstatus command.
rdqmstatus -m QM1
c. On the primary RDQM node, if the RDQM queue manager is started, stop it by using the endmqm -w or endmqm -i command.
d. On the primary RDQM node, determine the location of the data and log directories for the RDQM queue manager (use the methods described in steps 1b and 1c).
e. On the primary RDQM node, take backup of the contents of the RDQM queue manager data and log directories then delete, but not the directories themselves.
4. Restore the queue manager on the primary RDQM node: a. Unzip the queue manager data directory, then replace the files under the queue manager data directory.
for example: replace the file under '/var/mqm/vols/rdqm1/qmgr/qm1/'.
Also while replacing the queue manager data files, there is no need to replace the shared memory directories like '@app', '@qmgr', '@qmpersist'. But the '@ipcc' directory needs to be replaced since it includes AMQCLCHL.TAB and AMQRSYNA.DAT files.
b. Unzip the queue manager log directories, then replace the log files under queue manager log directory
for example: replace the files under '/var/mqm/vols/rdqm1/log/qm1/'
Note: While replacing the files, it is possible that some of the permissions might get altered.
We can run command like 'crtmqdir -f -a' to ensure check if all files exist and have correct set of permissions.
c. Ensure both queue manager data and log directories have correct set of permissions and ownerships.
d. Modify the qm.ini to provide the correct path of log files i.e, Log Path and other paths like exits if required.
e. Check state of queue manager with 'dspmq' command and 'rdqmstatus' commands.
f. Start the queue manager on the primary node.
g. Connect to the queue manager and update the value of the SSLKEYR queue manager attribute to specify the new location of the queue manager certificate repository. By default, the value of this attribute is set to the queue_manager_data_directory/ssl/key. The certificate repository must be located in the same location on each RDQM node. If the repository is not located in the replicated filesystem for the queue manager, then it must be copied to each node instead.
h. Review the IBM MQ object definitions for the queue manager and update the value of object attributes that reference changed network settings, the IBM MQ installation directory, or the queue manager data directory, including the following objects:
- Local IP addresses used by listeners (IPADDR attribute).
- Local IP addresses used by channels (LOCLADDR attribute).
- Local IP addresses defined for cluster-receiver channels (CONNAME attribute).
- Local IP addresses defined for communication information objects (GRPADDR attribute).
- System paths defined for process and service object definitions.
i. Stop and restart the queue manager to ensure the changes become effective.
j. Repeat step h for remote queue managers, plus equivalent settings for applications, that connect to the migrated queue manager, including:
- Channel connection names (CONNAME attribute).
- Channel authentication rules that restrict inbound connections from the queue manager based on its IP address or hostname.
- Client channel definition tables (CCDTs), domain name settings (DNS), network routing, or equivalent connection information.
k. Perform a managed failover of the queue manager to each RDQM node to ensure the required configuration has been established successfully.
References:https://www.ibm.com/docs/en/ibm-mq/9.2?topic=chr-migrating-queue-manager-become-ha-rdqm-queue-manager
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=availability-requirements-rdqm-ha-solution
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=solution-setting-up-passwordless-ssh
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=multiplatforms-installing-rdqm-replicated-data-queue-managers
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=availability-creating-ha-rdqm
Co-Author: Girish Dv (girishdv@in.ibm.com)