Deploying IBM(R) MQ Advanced RDQM HA on AWS
1. Overview
This article will help jumpstart integration teams to setup and run IBM MQ Replicated Data Queue Managers (RDQM) on AWS Cloud. This is an Active/Active/Active (AAA) scenario where all the 3 nodes will host Queue Managers. This sample can be extended to make use of MQ Clustering, and Uniform Clusters for message load balancing, and client connection load balancing. The scope of this article is to outline detailed instructions on how to deploy MQ RDQM HA Active/Active/Active Queue Managers on AWS Cloud. This article is based on John Colgrave's technical reference document https://github.com/ibm-messaging/mq-rdqm/tree/production_deployment/cloud/aws.
2. Architecture Diagram
This RDQM HA AAA PoC sample, will be deployed into 1 AWS Region having 3 Availability Zones (AZ). The Cloudformation stack template provided in reference link above contains all the segments to create the RDQM deployment stack.
3. Deployment to AWS
3.1 Create a VPC in AWS
For this sample PoC, I used the AWS Quick-Start-VPC CloudFormation template to create a VPC. As an AWS Administrator, login to AWS Console, navigate to Services > Management & Governance > CloudFormation > Stacks > Create Stack > With New Resources (Standard). Use the default options, and copy & paste https://aws-quickstart.s3.amazonaws.com/quickstart-aws-vpc/templates/aws-vpc.template.yaml URL for "AWS S3 Url" field.
Make sure you have the correct Region selected, and enter stack name (example Quick-Start-VPC-RDQM".
The RDQM sample deployment requires 3 Public Subnets, and 6 Private Subnets. Make sure to select 3 AZ's, and check CreateAdditionalPrivateSubnets option as below.
Follow the next screens, and create the Stack. If the stack completed successfully it should create 3 Public Subnets, and 6 Private Subnets. The Public Subnets will be used by the Public Load Balancer, and 3 Private Subnets will be used by the Private Load Balancer, and remaining 3 Private Subnets will be used for RDQM replication (see the topology diagram).
3.2 Capture VPC ID, Subnet ID's, and PrivateNetwork(B) IP's
Capture the subnet Id's of both Public, and Private subnets. These can be plugged into the Cloudformation template (rdqm.template.yaml).
Also, note down the CIDR IP's of PrivateNetwork 1B, 2B, and 3B. Based on the CIDR IP, we will update RDQM IPADDR field in setupRDQMInstance script.
3.3 Update rdqm.template.yaml
**** Edit rdqm.template.yaml and populate the default parameter values.
Parameters:
VpcId : Default: <vpc-id-value-captured-above>
PrivateSubnetA : Default: <Private subnet 1A-value-captured-above>
RdqmPrivateSubnetA : Default: <Private subnet 1B-value-captured-above>
RdqmAddressA : Default: <Private subnet 1B's IPv4 CIDR - 10.0.192.<nn>>. Note: If PrivateSubnet1B contains 10.0.128.0 then RdqmAddressA should follow CIDR 10.0.192.<nn> ex: 10.0.192.25.
Similarly polulate the below;
PrivateSubnetB : Default: <Private subnet 2A-ID value-captured-above>
RdqmPrivateSubnetB : Default: <Private subnet 2B-value-captured-above>
RdqmAddressB : Default: <Private subnet 2B - IPv4 CIDR - 10.0.200.<nn>>
PrivateSubnetC : Default: <Private subnet 3A-value-captured-above>
RdqmPrivateSubnetC : Default: <Private subnet 3B- id value-captured-above>
RdqmAddressC: Default: <Private subnet 3B - IPv4 CIDR - 10.0.208.<nn>>
3.4 Update setupRDQMInstance
**** Edit setupRDQMInstance
This step can be automated in the script, but for now need to hardcode the RDQM IP addresses in setupRdqmInstance.
Update IP addresses for functions configureNetworkForA, configureNetworkForB, configureNetworkForC.
configureNetworkForA
IPADDR=10.0.192.25 --- RdqmAddressA configured above in rdqm.template.yaml
10.0.200.25 via 10.0.192.1 dev eth1 --- Where, 10.0.200.25 is RdqmAddressB above, 10.0.192.1 is "Private Subnet 1B IpV4 CIDR + 1. Ex: 10.0.192.0 will be 10.0.192.1)
10.0.208.25 via 10.0.192.1 dev eth1 --- Where, 10.0.208.25 is RdqmAddressB above.
Similarly, configure other functions, and Save.
configureNetworkForB
IPADDR=10.0.200.25
10.0.192.25 via 10.0.200.1 dev eth1
10.0.208.25 via 10.0.200.1 dev eth1
configureNetworkForC
IPADDR=10.0.208.25
10.0.192.25 via 10.0.208.1 dev eth1
10.0.200.25 via 10.0.208.1 dev eth1
The scripts will be placed in RHEL Image Virtual machine /root/bin directory.
3.5 Create a keypair
In AWS Console, navigate to Key Pairs under "EC2 Features", create a key pair.
Download the keypair, and Save.
This key will be bsed by the Bastion Virtual Machine, RHEL Image Virtual Machine, and the 3 RDQM Virtual Machines.
3.6 Create a Bastion Virtual Machine
The Bastion Virtual Machine will use Public Subnet of your VPC. The RDQM virtual machines will be hosted inside the Private Network, hence you need to use the Bastion VM to SSH into RDQM instances.
You can use AWS quick-linux-bastion Cloudformation template "https://aws-quickstart.s3.amazonaws.com/quickstart-linux-bastion/templates/linux-bastion.template" OR you can create a Bastion Host manually. I created manually instead of the Cloudformation template. For this POC, I created one Bastion VM in one AZ.
From the AWS Console, navigate to EC2 > Instances > Launch Instances > Select Ubuntu Server 20.04 LTS > Review and Launch.
Select Instance Details, select the VPC created above, and Public Subnet1.
Create a new security group as below.
Use the Existing Keypair create above in section 3.2.
Wait for the instance to Provision.
3.7 Create RedHat Linux Virtual Machine
RDQM requires RHEL OS, hence we need to create a RHEL VM and install IBM MQ and supporting utilities. Once the all the required software is installed, this VM can be shutdown to build an AMI image which will be used later on by the Cloudformation template.
AWS Console > navigate to EC2 > Launch Instances > Select RHEL OS > Additional Details > Select VPC, and Public Subnet1.
Add additional storage for RDQM drbdpool.
Click Storage Section > add storage like below. You can add storage size based on your need.
Wait for the Virtual Machine to be provisoned.
3.8 IBM MQ Installation
After the RHEL VM is provisioned, proceed with MQ Installation.
sftp - Upload MQ Advanced
Once RHEL VM is ready, upload IBM MQ Advanced, and the scripts attached in this article.
sftp -i "rdqm-bastion.pem" ec2-user@ec2-xx-xxx-xx-xx.us-east-2.compute.amazonaws.com
sftp> put /Users/<username>/DownloadDirector/mq/IBM_MQ_9.2.5_LINUX_X86-64.tar.gz Note: You can download MQ Trial version if you do not have spare licenses.
sftp> put /Users/<username>/Downloads/setupRdqmInstance (updated above with RDQM IP's)
sftp> put /Users/<username>/Downloads/configureRDQM
quit
ssh - login to RHEL VM
ssh -i "rdqm-bastion.pem" ec2-user@ec2-xx-xxx-xx-xx.us-east-2.compute.amazonaws.com
Install the Pre-Req software
sudo -s
yum install bc
echo 'kernel.shmmax=268435456' >> /etc/sysctl.conf
echo 'vm.overcommit_memory=2' >> /etc/sysctl.conf
sysctl -p
echo 'fs.file-max=524288' >> /etc/sysctl.conf
echo '* hard nofile 10240' >> /etc/security/limits.conf
echo '* soft nofile 10240' >> /etc/security/limits.conf
echo 'root hard nofile 10240' >> /etc/security/limits.conf
echo 'root soft nofile 10240' >> /etc/security/limits.conf
sysctl -p
exit
sudo -i
Format the secondary storage (xvdb)
Run "lsblk" command to make sure to secondary storage is available.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /
xvdb 202:16 0 10G 0 disk
Run "fdisk" to create new partition and initialize the storage as below.
$ fdisk /dev/xvdb
Make sure xvdb1 is created by running "lsblk" again.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 10G 0 part /
xvdb 202:16 0 10G 0 disk
└─xvdb1 202:17 0 10G 0 part
Create physical volume, and volume group “drbdpool”
$ yum install lvm2. (contains pvcreate commands)
$ pvcreate /dev/xvdb1
$ vgcreate drbdpool /dev/xvdb1
$ pvs
$ vgs (make sure drbdpool is created).
Install Unzip Command.
yum install unzip
Note: the unzip command will be used in setupRDQMInstance script.
Finally, install MQ Advanced
$ sudo -s
$ cd /home/ce2-user
$ tar -xzvf IBM_MQ_9.2.5_LINUX_X86-64.tar.gz
$ rm *.gz
$ cd MQServer
$ ./mqlicense.sh -accept
Make sure the kmod version:
$ Advanced/RDQM/PreReqs/el8/kmod*/modver
kmod-drbd-9.1.5_4.18.0_305-1.x86_64.rpm
$ yum install Advanced/RDQM/PreReqs/el8/kmod-drbd-9/kmod-drbd-9.1.5_4.18.0_305-1.x86_64.rpm
$ yum install Advanced/RDQM/PreReqs/el8/drbd-utils-9/*
$ yum install Advanced/RDQM/PreReqs/el8/pacemaker-2/*
$ yum install MQSeriesGSKit* MQSeriesServer* MQSeriesRuntime* MQSeriesSamples* MQSeriesClient*
$ yum install Advanced/RDQM/MQSeriesRDQM*
$ /opt/mqm/bin/setmqinst -i -p /opt/mqm
147 of 147 tasks have been completed successfully.
'Installation1' (/opt/mqm) set as the primary installation.
$ usermod -a -G haclient,mqm ec2-user
$ vi /home/ec2-user/.bash_profile (add below lines)
. /opt/mqm/bin/setmqenv -s
set -o vi
# update selinux config file
vi /etc/selinux/config
SELINUX=permissive
Logoff and logback as ec2-user
$ ssh -i "rdqm-bastion.pem" ec2-user@ec2-xx-xxx-xx-xx.us-east-2.compute.amazonaws.com
$ mqconfig
Granting sudo access to mqm
To grant the required sudo access to the mqm user, as root
vi /etc/sudoers.d/mqm
mqm ALL=(root) NOPASSWD: /opt/mqm/bin/crtmqm, /opt/mqm/bin/dltmqm, /opt/mqm/bin/rdqmadm, /opt/mqm/bin/rdqmstatus
3.9 Create AWS AMI
Now, we are now ready to create AMI based on RHEL VM that we configured with MQ Advanced, and the RDQM scripts.
From AWS Console > Instances > Select RHEL VM > STOP the instance.
AWS Console > Instances > Select RHEL VM > Actions > image and template > Create Image.
Save the AMI ID, will be used in the Cloudformation template rdqm.template.yaml.
3.9 Deploy the Cloudformation Stack
Deploy the modified template (rdqm.template.yaml).
AWS Console > Cloudformation > Create Stack > With new Resources (standard) > Upload a template file > rdqm.template.yaml.
Paste your RHEL VM's AMI image that you have created above.
Select the "key pair" that you have created earlier.
Finish creating the Cloudformation template. Make sure the stack is created successfully. When the stack is created successfully you should see the 3 VM's created.
3.10 Update LoadBalancer configuration
Update both Public, and Private load balancers of your VPC, and enable "Cross-zone load balancing.
AWS Console > Navigate to "Load Balancers".
Select "PrivateListenerLoadBalancer" > Navigate to Attributes section under the Description tab > Edit Attributes > Enable Cross-zone load balancing.
Repeat the same for "PublicListenerLoadBalancer".
4. Create RDQM Queue Managers
Open 3 Terminal or command line windows.
SSH in to the bastion virtual machine from each terminal window.
From the bastion, ssh into each RDQM Virtual Machine ( Locate SSH Configuration through EC2 > RDQM VM > Connect > SSH Client).
RDQM VM1
$ sudo -s
$ su - mqm
$ crtmqm -sx -p 1501 RDQM1
RDQM VM2
$ sudo -s
$ su - mqm
$ crtmqm -sx -p 1502 RDQM2
RDQM VM3
$ sudo -s
$ su - mqm
$ crtmqm -sx -p 1503 RDQM3
Make sure the queue managers are Running.
[mqm@InstanceA ~]$ rdqmstatus
Node: InstanceA
OS kernel version: 4.18.0-305
DRBD OS kernel version: 4.18.0-305
DRBD version: 9.1.5
DRBD kernel module status: Loaded
Queue manager name: RDQM1
Queue manager status: Running
HA status: Normal
HA current location: This node
HA preferred location: This node
HA blocked location: None
Queue manager name: RDQM2
Queue manager status: Running elsewhere
HA status: Normal
HA current location: InstanceB
HA preferred location: InstanceB
HA blocked location: None
Queue manager name: RDQM3
Queue manager status: Running elsewhere
HA status: Normal
HA current location: InstanceC
HA preferred location: InstanceC
HA blocked location: None
Congratulations, you have got a running RDQM HA Active/Active/Active configuration on AWS.
5. Testing
You can test from the bastion virtual machine using the PrivateListenerLoadBalancer DNS (OR) test from your PC through the PublicListenerLoadBalancer DNS address.
Configure a queue manager to Put Messages, for testing purpose disabling the CHLAUTH security.
RDQM VM1
Login to RDQM VM1 through the Bastion VM.
Create user in RDQM VM1 (ID must be same as your PC login ID - MQCSP will need use this ID).
$ sudo adduser -G mqm <your-userid>
-- Create MQ Definitions
$ runmqsc RDQM1
ALTER QMGR CHLAUTH(DISABLED)
ALTER QMGR CONNAUTH(' ')
DEFINE CHANNEL(RDQM.SVRCONN) CHLTYPE(SVRCONN)
DEFINE QLOCAL(TEST.IN)
Testing from your PC - Connect to RDQM1 Queue Manager through the Public Load Balancer
Using amqsphac sample program
Use "amqsphac" to put messages into the Queue from your PC (make sure you have MQClient, and Samples installed locally).
$ export MQSERVER="RDQM.SVRCONN/TCP/PublicListenerLoadBalancer-xxxxx.elb.us-east-2.amazonaws.com(1501)"
$ amqsphac TEST.IN RDQM1
MacBook-Pro-2:~ myuserid$ amqsphac TEST.IN RDQM1
Sample AMQSPHAC start
target queue is TEST.IN
message <Message 1>
message <Message 2>
Using MsgManager Java Tool - (Windows, Unix, Mac workstations)
Download Link: https://community.ibm.com/community/user/integration/viewdocument/mq-message-manager-update?CommunityKey=d974a63b-37b1-4759-8c62-a9d4a6f71297
6. Acknowlegment
I would like to thank John Colgrave (Disaster Recovery and High Availability Architect, IBM MQ), and Joseph Jodl (Advisory Technical Specialist, Automation IBM Technology, US National Market - Midwest) for helping to understand RDQM HA in AWS and reviewing the article. Their feedback was extremely helpful for fine-tuning the article.