In IBM MQ 9.1.2 distributed platforms, application balancing functionality was added for use with uniform clusters. With this feature re-connectable clients are automatically moved from a queue manager (QM) with more connections to a QM with less connections.
For an overview of application rebalancing within uniform clusters see the following blogs:
Prior to IBM MQ 9.1.3, application status could only be monitored by querying connections and filtering by APPLTAG. From MQ 9.1.3 on distributed, a new MQSC command DISPLAY APSTATUS and associated PCF command MQCMD_INQUIRE_APPL_STATUS have been added to display user application status (system applications are not displayed).
These commands are primarily targeted for QMs in a uniform cluster although they can be used elsewhere, such as a standalone QM or QM that is part of a regular cluster if preparing to convert into a uniform cluster for example.
This blog focuses on the MQSC command DISPLAY APSTATUS which I will refer to by it’s synonym DIS APS. There are three modes of DIS APS, each defined by the type parameters ‘appl’, ‘qmgr’ & ‘local’. Each mode provides an individual perspective of the uniform cluster as shown below.
DIS APS TYPE(APPL)
- Gives a uniform cluster wide view of the applications connected
DIS APS TYPE(QMGR)
- Gives a view of where the applications are connected across a uniform cluster by queue manager
- Can also see if the remote QMs are talking to us or not
DIS APS TYPE(LOCAL)
- Gives a view of the local instances of applications connected to this QM
Note: for a full breakdown of DIS APS parameters see the knowledge centre documentation.
With these perspectives we can ask and answer the following questions:
What is running in my uniform cluster?
- Show summary counts of the applications instances across the whole uniform cluster
Is my application balanced correctly?
- Show me my applications and how they are spread across the members of the uniform cluster
What’s connected here and are they movable (re-connectable)?
- Show me the application instances on a specific QM
Why wont a particular application instance move?
- Show instance specific details relating to balancing
We will set up a uniform cluster and deploy various configurations of re-connectable and non re-connectable applications. With each configuration we will demonstrate how DIS APS can be used to query the state of applications, application instances, connections and QMs within the uniform cluster to answer the questions above. To achieve this we will complete the following tasks:
Task 1: Create a demo test environment
1.1 Set up terminals
1.2 Export environment variables
-
Run the following commands in terminal 4:
export MQAPPLNAME="MY_APP_1" to name all subsequent applications deployed in this terminal as ‘MY_APP_1’.
export MQCCDTURL="file:///tmp/ccdt.json" to route all subsequent applications deployed in this terminal to our CCDT file.
export PATH=$PATH:/opt/mqm/bin:/opt/mqm/samp/bin to add MQ binaries to the PATH variable.
1.4 Set up uniform cluster
-
In terminal 4 create a file uniclus.ini and paste in the following:
AutoCluster:
Repository2Conname=localhost(1414)
Repository2Name=QM1
Repository1Conname=localhost(1415)
Repository1Name=QM2
ClusterName=UNICLUS
Type=Uniform
-
Create a file uniclus.mqsc and paste in the following:
define channel('+AUTOCL+_+QMNAME+') chltype(clusrcvr) trptype(tcp) conname('+CONNAME+') cluster('+AUTOCL+') replace
define channel(CLIENTCHL) chltype(svrconn) trptype(tcp) replace
set chlauth(CLIENTCHL) type(addressmap) address(*) usersrc(channel)
set chlauth(CLIENTCHL) type(blockuser) userlist('nobody')
alter authinfo(system.default.authinfo.idpwos) authtype(idpwos) chckclnt(optional)
refresh security
Note:each queue manager will have a svrconn defined.
-
Create & start 3 QMs: QM1, QM2 & QM3
crtmqm -p 1414 -ii uniclus.ini -ic uniclus.mqsc -iv "CONNAME=localhost(1414)" QM1 && strmqm QM1
crtmqm -p 1415 -ii uniclus.ini -ic uniclus.mqsc -iv "CONNAME=localhost(1415)" QM2 && strmqm QM2
crtmqm -p 1416 -ii uniclus.ini -ic uniclus.mqsc -iv "CONNAME=localhost(1416)" QM3 && strmqm QM3
1.5 Create JSON CCDT
-
Create a file of name ‘ccdt.json’ and paste in the following JSON code:
{
"channel": [
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1415 } ], "queueManager": "QMGROUP" },
"connectionManagement": { "clientWeight": 1, "affinity": "none" },
"type": "clientConnection"
},
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1415 } ], "queueManager": "QM1" },
"type": "clientConnection"
},
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1416 } ], "queueManager": "QMGROUP" },
"connectionManagement": { "clientWeight": 1, "affinity": "none" },
"type": "clientConnection"
},
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1416 } ], "queueManager": "QM2" },
"type": "clientConnection"
},
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1417 } ], "queueManager": "QMGROUP" },
"connectionManagement": { "clientWeight": 1, "affinity": "none" },
"type": "clientConnection"
},
{
"name": "CLIENTCHL",
"clientConnection": { "connection": [ { "host": "localhost", "port": 1417 } ], "queueManager": "QM3" },
"type": "clientConnection"
}
]
}
Note:
For this demo we will place this file in the /tmp directory, although in practice this file can be located anywhere.
We have defined each CLNTCONN twice per QM, first as a member of a QM Group and second as an independent connection to its respective QM.
The former definition allows re-connectable application instances to load balance across all QMs in the uniform cluster, while the latter is used to couple application instances to specific QMs.
In the event of a failed rebalance an application instance using the former definition will reconnect to any queue manager within the group, whereas with the latter definition an instance will only reconnect to the specified queue manager.
We will take advantage of the latter to initially set the application into an unbalanced state by deploying all the application instances to QM1, we will then see the instances rebalance by issuing ourDIS APS commands.
To enable initial load balancing for the QM group connections, connectionManagement objects are defined with clientWeight & affinity attributes set to 1 & none respectively.
We now have our environment set up! Let’s see how we can use DIS APS to inspect our uniform cluster…
Task 2: What is running in my uniform cluster?
2.1 Deploy re-connectable applications
-
In terminal 4, repeat the following command 9 times: amqsghac SYSTEM.DEFAULT.LOCAL.QUEUE QM1 > /dev/null 2>&1 &
Note:
We redirect stdout & stderr to null and fork the process so we can continue working in the same terminal.
Sample program amqsghac is a get application with the re-connectable flag set.
As we have deployed all application instances to QM1, the uniform cluster will initially be in an unbalanced state.
2.2 Monitor application rebalancing
-
In any terminal 1 through 3, run the command DIS APS(MY_APP_1) TYPE(APPL) and observe the output, initially you will see BALANCED(NO):
APPLNAME(MY_APP_1) CLUSTER(MY_CLUSTER)
COUNT(9) MOVCOUNT(9)
BALANCED(NO)
Note: DIS APS with TYPE(APPL) & TYPE(QMGR) parameters can be run from any QM in the uniform cluster as they offer a cluster-wide perspective, whereas TYPE(LOCAL) offers a perspective of the specific QM targeted.
-
In any terminal 1 through 3, run the command DIS APS(MY_APP_1) TYPE(QMGR) and observe the output.
Initially you will see BALSTATE(LOW) & COUNT(0) on QMs 2 & 3, while on QM1 you will see BALSTATE(HIGH) & COUNT(9):
APPLNAME(MY_APP_1) ACTIVE(YES)
COUNT(9) MOVCOUNT(9)
BALSTATE(HIGH) LMSGDATE(2019-07-20)
LMSGTIME(23:52:07) QMNAME(QM1)
QMID(QM1_2019-07-19_10.19.43)
APPLNAME(MY_APP_1) ACTIVE(YES)
COUNT(0) MOVCOUNT(0)
BALSTATE(LOW) LMSGDATE(2019-07-20)
LMSGTIME(23:51:16) QMNAME(QM2)
QMID(QM2_2019-07-19_10.19.56)
APPLNAME(MY_APP_1) ACTIVE(YES)
COUNT(0) MOVCOUNT(0)
BALSTATE(LOW) LMSGDATE(2019-07-20)
LMSGTIME(23:52:06) QMNAME(QM3)
QMID(QM3_2019-07-19_10.20.25)
After a few minutes when all application instances have rebalanced, repeat the command and you will see BALSTATE(OK) and COUNT(3) on all QMs.
-
In any terminal 1 through 3, run again the command DIS APS(MY_APP_1) TYPE(APPL) and observe the output, you will now see BALANCED(YES).
Note:
BALANCED in APPL context is the state of the application in the uniform cluster (YES/NO), while BALSTATE in QMGR context is the state of application instances per QM (lOW/HIGH/OK).
The rebalance latency is governed by the rate of requests to transfer applications which is currently one request per minute
Information transferred between QMs currently occurs every 10 seconds, as a result you may have to wait up to 20 seconds for correct consensus on state across the QMs.
2.3 Deploy another unique set of named application instances
-
In terminal 4, runexport MQAPPLNAME="MY_APP_2" To name all subsequent applications deployed in this terminal as ‘MY_APP_2’.
-
In terminal 4, repeat the following command 6 times:amqsghac SYSTEM.DEFAULT.LOCAL.QUEUE *QMGROUP > /dev/null 2>&1 &
Note:as we have used the QM group client connections with load balancing attributes defined, the time to realise balanced state will be greatly reduced, if not instantaneous.
2.4 Show summary counts of application instances across the whole uniform cluster
-
In terminals 1 through 3, run the command DIS APS(*Y_APP_*) TYPE(APPL) and observe the output. Each QM will report the same global perspective. You will now see two entries, one for ‘MY_APP_1’, the other for ‘MY_APP_2’:
APPLNAME(MY_APP_2) CLUSTER(MY_CLUSTER)
COUNT(6) MOVCOUNT(6)
BALANCED(YES)
APPLNAME(MY_APP_1) CLUSTER(MY_CLUSTER)
COUNT(9) MOVCOUNT(9)
BALANCED(YES)
Note:we have used multiple wildcards to filter application name.
-
In any terminal 1 through 3, run the command DIS APS(*APP_2) TYPE(APPL) and observe the output, you will now see one filtered entry for ‘MY_APP_2’:
APPLNAME(MY_APP_2) CLUSTER(MY_CLUSTER)
COUNT(6) MOVCOUNT(6)
BALANCED(YES)
2.5 Filter all applications on a specific cluster
-
In any terminal 1 through 3, run the command DIS APS(*) TYPE(APPL) WHERE(CLUSTER EQ MY_CLUSTER) and observe the output, you will see our applications as expected:
APPLNAME(MY_APP_2) CLUSTER(MY_CLUSTER)
APPLNAME(MY_APP_1) CLUSTER(MY_CLUSTER)
Note: a partial repository QM that is a member of a uniform cluster can also be a member of another regular cluster, but not another uniform cluster. Had we set up intersecting clusters we could filter applications that do not belong to our uniform cluster, but that is outside the scope of this demo.
Task 3: Is my application balanced correctly?
3.1 Filter unbalanced applications
-
In any terminal 1 through 3, run the command DIS APS(*) TYPE(APPL) WHERE(BALANCED EQ NO) and observe the following output from which we can infer that all applications are balanced:
AMQ8933I: Application status not found.
Note: we have used a single wildcard to filter all application names.
3.2 Force ‘MY_APP_2’ into an unbalanced state
-
In terminal 4, repeat the following command 6 times: amqstrgc SYSTEM.DEFAULT.LOCAL.QUEUE QM1 > /dev/null 2>&1 &
Note:
Application amqstrgc does not have the re-connectable flag set and has unlimited wait time.
Environment variable MQAPPLNAME was previously set to ‘MY_APP_2’, so these non re-connectable instances will have the same name as our re-connectable instances we deployed in step 2.3.
-
As the uniform cluster attempts to rebalance ‘MY_APP_2’, the 6 original re-connectable application instances will balance themselves across QM2 and QM3 while the 6 non re-connectable application instances we just deployed will sit on QM1, as a result when rebalancing has completed our application remains unbalanced. You can monitor this process by running the command DIS APS(*APP_2) TYPE(QMGR) in any terminal 1 through 3.
3.3 Filter unbalanced applications
Task 4: What’s connected here and are they movable (re-connectable)?
4.1 Investigate
Now if we were to forget that we had forced an unbalanced state as in step 3.2, and wanted to investigate why we had found an unbalanced application as in step 3.3, we could do the following:
-
In any terminal 1 through 3, run the command DIS APS(MY_APP_2) TYPE(QMGR) and observe the output.
APPLNAME(MY_APP_2) ACTIVE(YES)
COUNT(6) MOVCOUNT(0)
BALSTATE(HIGH) LMSGDATE(2019-07-20)
LMSGTIME(22:49:41) QMNAME(QM1)
QMID(QM1_2019-07-19_10.19.43)
APPLNAME(MY_APP_2) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(LOW) LMSGDATE(2019-07-20)
LMSGTIME(22:49:04) QMNAME(QM3)
QMID(QM3_2019-07-19_10.20.25)
APPLNAME(MY_APP_2) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(LOW) LMSGDATE(2019-07-20)
LMSGTIME(22:49:15) QMNAME(QM2)
QMID(QM2_2019-07-19_10.19.56)
As QM1 has properties MOVCOUNT(0) & COUNT(6), we may ask ourselves why these 6 application instances have not rebalanced…
Task 5: Why wont a particular application instance move?
5.1 Show me the application instances on a specific QM
-
In terminal 1, run the command DIS APS(MY_APP_2) TYPE(LOCAL) and observe the output:
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01841821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01821821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01811821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01801821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01831821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
APPLNAME(MY_APP_2)
CONNTAG(MQCTA39A315D01851821QM1_2019-07-19_10.19.43MY_APP_2)
CONNS(1) IMMREASN(NOTRECONN)
IMMCOUNT(0) IMMDATE( )
IMMTIME( ) MOVABLE(NO)
5.2 Show instance specific details relating to balancing
-
From the output of 5.1 We can see that all ‘MY_APP_2’ application instances on QM1 have the values MOVABLE(NO) & IMMREASN(NOTRECONN), these values inform us that these instances are considered un-moveable because they are non re-connectable.
Note:you can use the CONNTAG field as a filter to the MQSC DISPLAY CONN command to see individual QM connections from that instance, which can be useful to determine the root cause of why an application instance is considered non re-connectable. For an example see Monitoring application balancing.
Task 6: Is my uniform cluster healthy?
-
In terminal 4, run the command endmqm -i QM1 to simulate an unplanned outage and wait a couple of minutes while the uniform cluster rebalances and achieves consensus.
-
In terminal 2 or 3, run the command DIS APS(MY_APP*) TYPE(QMGR) and observe the result:
APPLNAME(MY_APP_2) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(OK) LMSGDATE(2019-08-01)
LMSGTIME(08:22:55) QMNAME(QM2)
QMID(QM2_2019-08-01_07.57.33)
APPLNAME(MY_APP_2) ACTIVE(NO)
COUNT(0) MOVCOUNT(0)
BALSTATE(NOTAPPLIC) LMSGDATE(2019-08-01)
LMSGTIME(08:17:23) QMNAME(QM1)
QMID(QM1_2019-08-01_07.57.30)
APPLNAME(MY_APP_2) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(OK) LMSGDATE(2019-08-01)
LMSGTIME(08:22:24) QMNAME(QM3)
QMID(QM3_2019-08-01_07.57.36)
APPLNAME(MY_APP_1) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(OK) LMSGDATE(2019-08-01)
LMSGTIME(08:22:55) QMNAME(QM2)
QMID(QM2_2019-08-01_07.57.33)
APPLNAME(MY_APP_1) ACTIVE(NO)
COUNT(0) MOVCOUNT(0)
BALSTATE(NOTAPPLIC) LMSGDATE(2019-08-01)
LMSGTIME(08:17:23) QMNAME(QM1)
QMID(QM1_2019-08-01_07.57.30)
APPLNAME(MY_APP_1) ACTIVE(YES)
COUNT(3) MOVCOUNT(3)
BALSTATE(OK) LMSGDATE(2019-08-01)
LMSGTIME(08:22:24) QMNAME(QM3)
QMID(QM3_2019-08-01_07.57.36)
QM1 has the property ACTIVE(NO), this tells us that there is a problem with QM1 and we should investigate.
Running endmqmwith the-i flag has forced an immediate shutdown of QM1, as a consequence all application instances on QM1, both re-connectable and non re-connectable have been lost.
This can be verified by running the command DIS APS(MY_APP*) TYPE(APPL) on terminal 2 or 3 and observing the output:
APPLNAME(MY_APP_2) CLUSTER(MY_CLUSTER)
COUNT(6) MOVCOUNT(6)
BALANCED(YES)
APPLNAME(MY_APP_1) CLUSTER(MY_CLUSTER)
COUNT(6) MOVCOUNT(6)
BALANCED(YES)
You can see that we have lost the 3 of the 9 re-connectable instances of ‘MY_APP_1’ and the 6 of the 12 non re-connectable MY_APP_2 instances, leaving us with 6 instances of each application.
-
If we had run the command endmqm with the -r flag to simulate a planned outage of QM1, the re-connectable ‘MY_APP_1’ instances on QM1 would have rebalanced across QM2 & QM3, wheras the non re-connectable ‘MY_APP_2’ instances would be lost. We will leave this as an exercise for the reader!
Note: uniform clusters can be configured to implement high availability, although this is outside of the scope of this demo.
Summary
During this exercise, we have used DIS APS to:
Closing
We’ve now gone through the basic functionality of using DIS APS, demonstrating the ability to monitor and manage your connected application instances. Now that you have a template installation, please feel free to reference the documentation and experiment with DIS APS in more detail. Also see Application balancing trouble shooting for a list of symptoms and solutions associated with application balancing.
As uniform clusters, application balancing and DIS APS features are works in progress, the algorithm for rebalancing and inter-cluster communication timing windows are subject to change, as a consequence the behaviour of DIS APS may vary over time. Look out for upcoming blogs on further developments, and please post any thoughts on improvements or additional features that you would like to see.
#MQ