• We have a UC where there are multiple (># of cluster members) receiving client applications that remain connected, waiting to receive messages from a queue.  The cluster will automatically handle balancing these applications in a way that balances the connections to each QM.
• The message sender is a batch process that wakes up periodically, connects to the cluster and sends a burst of messages.  It is a single instance that connects to MQ when it wakes and then disconnects once its work is done.
• Each message is independent of the rest so it doesn't matter which receiver process a message, or in what order they get handled.  The message pattern is point-to-point.  No reply needed.

• Is this the right approach, or is there a better way to handle this in a UC?
• Should my version of AMQSCLM be run on each of the MQ servers?  Or, is it better to run this in a separate container (Kubernetes)?
• Any issues with modifying/extending AMQSCLM (e.g., licensing) for this purpose?

• When more than one remote instance of a destination remains and all channels to that destination have CLWLWGHT set to the default setting of 50, the least recently used channel is chosen. This approximately equates to a round-robin style of workload balancing when multiple remote instances exist.

Yes, I think you've found the right sources to be looking at and your high level understanding is correct.  For a clustered queue the 'destination binding' is never fixed at the connect time - it will be at OPEN, PUT or (less commonly) between message groups.

As you've noted, there is not (currently) any mechanism to use target queue depth as a factor in the balancing algorithm no, only information readily available at the sending queue manager side is considered.  This has occasionally been mooted as a product enhancement in the past but is not one which has received a great deal of support/pressure from the community so hasn't risen to the 'top of the pile' to date.

The only other thing you may have missed which is important to be aware of is that the balancing is not 'per queue' but 'per channel' (or effectively 'per queue manager').  So if the scenario was exactly as described, with no other queues in the mix, then yes you would expect to see (more or less) exactly 50:50 balancing.  However, once other queues and applications are involved (some of which might have different balancing requirements, such as using BIND_ON_OPEN), then you might get slightly different results.  In practice, in most real world scenarios you probably wouldn't notice the difference but it does sometimes catch people out.

*
* Channel clients use to connect.
DEFINE CHANNEL('CLNT.QC9') CHLTYPE(SVRCONN) SHARECNV(1) REPLACE
SET CHLAUTH('CLNT.QC9') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED) DESCR('Allows connection via APP channel') ACTION(REPLACE)
*
* Local queue definitions:

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) REPLACE

     8 : dis chstatus(D*) 
AMQ8417I: Display Channel Status details.
   CHANNEL(DEVUC_QC91)                     CHLTYPE(CLUSRCVR)
   CONNAME(172.19.0.4)                     CURRENT
   RQMNAME(QC90)                           STATUS(RUNNING)
   SUBSTATE(RECEIVE)                    
AMQ8417I: Display Channel Status details.
   CHANNEL(DEVUC_QC90)                     CHLTYPE(CLUSSDR)
   CONNAME(172.19.0.4(9414))               CURRENT
   RQMNAME(QC90)                           STATUS(RUNNING)
   SUBSTATE(MQGET)                         XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE)

 let od = new mq.MQOD();
 od.ObjectName = queueName;
 od.ObjectType = MQC.MQOT_Q;
 let openOptions = MQC.MQOO_OUTPUT | MQC.MQOO_BIND_NOT_FIXED;

 mq.Open(hConn, od, openOptions, function (err, hObj) {
  ...
  }

 let mqmd = new mq.MQMD(); 
 mqmd.Persistence = false;
 mqmd.PutApplName = pgmName;

 let pmo = new mq.MQPMO();
 pmo.Options = MQC.MQPMO_NO_SYNCPOINT |
      MQC.MQPMO_NEW_MSG_ID |
      MQC.MQPMO_NEW_CORREL_ID;
 mq.PutSync(hObj, mqmd, pmo, msg, function (err) {
   ...
  }

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) CLUSTER(DEVUC) REPLACE

*
* Channel clients use to connect.
DEFINE CHANNEL('CLNT.QC9') CHLTYPE(SVRCONN) SHARECNV(1) REPLACE
SET CHLAUTH('CLNT.QC9') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED) DESCR('Allows connection via APP channel') ACTION(REPLACE)
*
* Local queue definitions:
DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) REPLACE

     8 : dis chstatus(D*) 
AMQ8417I: Display Channel Status details.
   CHANNEL(DEVUC_QC91)                     CHLTYPE(CLUSRCVR)
   CONNAME(172.19.0.4)                     CURRENT
   RQMNAME(QC90)                           STATUS(RUNNING)
   SUBSTATE(RECEIVE)                    
AMQ8417I: Display Channel Status details.
   CHANNEL(DEVUC_QC90)                     CHLTYPE(CLUSSDR)
   CONNAME(172.19.0.4(9414))               CURRENT
   RQMNAME(QC90)                           STATUS(RUNNING)
   SUBSTATE(MQGET)                         XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE)

 let od = new mq.MQOD();
 od.ObjectName = queueName;
 od.ObjectType = MQC.MQOT_Q;
 let openOptions = MQC.MQOO_OUTPUT | MQC.MQOO_BIND_NOT_FIXED;

 mq.Open(hConn, od, openOptions, function (err, hObj) {
  ...
  }

 let mqmd = new mq.MQMD(); 
 mqmd.Persistence = false;
 mqmd.PutApplName = pgmName;

 let pmo = new mq.MQPMO();
 pmo.Options = MQC.MQPMO_NO_SYNCPOINT |
      MQC.MQPMO_NEW_MSG_ID |
      MQC.MQPMO_NEW_CORREL_ID;
 mq.PutSync(hObj, mqmd, pmo, msg, function (err) {
   ...
  }

On your first scenario, if half your messages were to go via QC90 and the other half via QC91, it wouldn't be along the lines you have drawn on your picture. Your "sending client" application is, I assume, making a connection to one of QC90 or QC91 - that's the dotted line in your picture. Then, having connected to a queue manager, let's assume QC90, the application proceeds to put messages. These puts will either resolve to a local instance of a clustered queue or a remote instance of a cluster queue, in the latter case they land up on the SYSTEM.CLUSTER.TRANSMIT.QUEUE ready to be delivered to another queue manager. As you learned in earlier responses to this thread, this depends on the queue being clustered (i.e. having a non-blank value in the CLUSTER keyword) and how you have the CLWLUSEQ attribute set. The same is true for this scenario, except that, the use case for sending half of the messages from QC90 to QC91 in order to then have them be routed to QS90 seems less clear. This is a case where CLWLUSEQ(LOCAL) would be far more appropriate don't you think?

Your second scenario is possible, but only if QS90 is also in the cluster with QC90 and QC91. Because QS90 is not in the cluster, it does not have access to the knowledge of where the clustered queues reside and so cannot route messages to all hosted queues because it does not know them. If it was in the cluster, the remote queue definition on QS90 could leave the RQMNAME attribute blank, and messages put using that remote queue definition would act in the same way as if you had done a put to queue name on blank qmgr - that allows the cluster choosing algorithm to get in there and pick. I think you'd have to change the DEFBIND option on the remote queue definition if the application was connecting to a queue manager outside of the cluster as it's MQOO option wouldn't carry through.

Are you just trying out different scenarios as a learning exercise, or do you have a problem to solve that we could help with?

Cheers,
Morag

Original Message:
Sent: Fri January 07, 2022 03:52 PM
From: Jim Creasman
Subject: Balancing messages in a uniform cluster based on queue depth

Now that I have a clustered queue defined properly I've been playing around with some different scenarios involving remote queue definitions.  In the first scenario I have a the local queue defined on a non-clustered qmgr and the remote queue definitions on the clustered qmgrs.  A picture is the best way to describe.


When my sending client sends a batch of messages and BIND_NOT_FIXED is used, do half of these go through QC90 and the other half go through QC91?  Regardless, all messages are forwarded to the local queue on QS90 as expected.  I'm just curious as to whether BIND_NOT_FIXED applies in this case.

My next question is, can I flip this script as in the following diagram?  This time the local queue definition is in the cluster (QC90 & QC91).  The client is sending to a queue that is defined as remote on qmgr, QS90.  If the client were connected directly to a cluster qmgr and PUT the messages with BIND_NOT_FIXED then the messages would be divided between the two queues.  Is there an option similar to BIND_NOT_FIXED that can be specified when the remote queue on QS90 is created such that it acts in the same way?  In other words, is it possible for the client to put 100 messages to the remote queue and have 50 end up on each of the two clustered queues?


Thanks,
Jim

------------------------------
Jim Creasman
------------------------------

DEFINE QREMOTE('MQDEV.QUEUE.V1') RNAME('MQDEV.QUEUE.V1') RQMNAME('DEVUC.RQMA') XMITQ('QC90') REPLACE

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) CLUSTER(DEVUC) DEFBIND(NOTFIXED) REPLACE

DEFINE QREMOTE('DEVUC.RQMA') RNAME('') RQMNAME('') XMITQ('') REPLACE

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) CLUSTER(DEVUC) DEFBIND(NOTFIXED) REPLACE

Original Message:
Sent: Fri January 07, 2022 03:52 PM
From: Jim Creasman
Subject: Balancing messages in a uniform cluster based on queue depth

Now that I have a clustered queue defined properly I've been playing around with some different scenarios involving remote queue definitions.  In the first scenario I have a the local queue defined on a non-clustered qmgr and the remote queue definitions on the clustered qmgrs.  A picture is the best way to describe.


When my sending client sends a batch of messages and BIND_NOT_FIXED is used, do half of these go through QC90 and the other half go through QC91?  Regardless, all messages are forwarded to the local queue on QS90 as expected.  I'm just curious as to whether BIND_NOT_FIXED applies in this case.

My next question is, can I flip this script as in the following diagram?  This time the local queue definition is in the cluster (QC90 & QC91).  The client is sending to a queue that is defined as remote on qmgr, QS90.  If the client were connected directly to a cluster qmgr and PUT the messages with BIND_NOT_FIXED then the messages would be divided between the two queues.  Is there an option similar to BIND_NOT_FIXED that can be specified when the remote queue on QS90 is created such that it acts in the same way?  In other words, is it possible for the client to put 100 messages to the remote queue and have 50 end up on each of the two clustered queues?


Thanks,
Jim

------------------------------
Jim Creasman
------------------------------

With those definitions, and QS90 not in the cluster, all the messages are going via QC90. Some will then make a second hop to QC91. Just checking that was what you wanted?

Cheers,
Morag

Original Message:
Sent: Thu January 13, 2022 02:26 PM
From: Jim Creasman
Subject: Balancing messages in a uniform cluster based on queue depth

I have the second scenario working now using the following definitions:

QS90 (single MQ server):

DEFINE QREMOTE('MQDEV.QUEUE.V1') RNAME('MQDEV.QUEUE.V1') RQMNAME('DEVUC.RQMA') XMITQ('QC90') REPLACE

QC90 (clustered):

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) CLUSTER(DEVUC) DEFBIND(NOTFIXED) REPLACE

DEFINE QREMOTE('DEVUC.RQMA') RNAME('') RQMNAME('') XMITQ('') REPLACE

QC91 (clustered):

DEFINE QLOCAL('MQDEV.QUEUE.V1') CLWLUSEQ(ANY) CLUSTER(DEVUC) DEFBIND(NOTFIXED) REPLACE

Now, when I put 10 messages to MQEV.QUEUE.V1 on QS90 I see 5 of these show up on QC90 and the other 5 on QC91.  QED

Thanks everyone for your help.

Jim

------------------------------
Jim Creasman
------------------------------


Original Message:
Sent: Fri January 07, 2022 03:52 PM
From: Jim Creasman
Subject: Balancing messages in a uniform cluster based on queue depth

Now that I have a clustered queue defined properly I've been playing around with some different scenarios involving remote queue definitions.  In the first scenario I have a the local queue defined on a non-clustered qmgr and the remote queue definitions on the clustered qmgrs.  A picture is the best way to describe.


When my sending client sends a batch of messages and BIND_NOT_FIXED is used, do half of these go through QC90 and the other half go through QC91?  Regardless, all messages are forwarded to the local queue on QS90 as expected.  I'm just curious as to whether BIND_NOT_FIXED applies in this case.

My next question is, can I flip this script as in the following diagram?  This time the local queue definition is in the cluster (QC90 & QC91).  The client is sending to a queue that is defined as remote on qmgr, QS90.  If the client were connected directly to a cluster qmgr and PUT the messages with BIND_NOT_FIXED then the messages would be divided between the two queues.  Is there an option similar to BIND_NOT_FIXED that can be specified when the remote queue on QS90 is created such that it acts in the same way?  In other words, is it possible for the client to put 100 messages to the remote queue and have 50 end up on each of the two clustered queues?


Thanks,
Jim

------------------------------
Jim Creasman