MQ

 View Only
Expand all | Collapse all

MQ subscriptions with multiple client instances

  • 1.  MQ subscriptions with multiple client instances

    IBM Champion
    Posted Mon September 20, 2021 03:22 PM
    I'm trying to understand how to deal with the scenario where I have a subscriber to a topic, and the subscriber is running in a service that may exist as multiple instances (e.g., as in the case of being scaled up for handling extra load).  Normally, each subscriber receives it's own copy of a message.  In this case if I have say 3 instances of the service running I don't want each one to get their own copy.  Instead, I would like to distribute the subscription across the set of instances, similar to how HTTP requests are handled by a load balancer.  

    One way I can see to do this is to define an administrative subscription on the queue manager that receives messages published to the topic, and then puts these on a queue (not topic).  Then, change the clients to connect to the queue.  Each instance of the client would get a message, thus preventing the same message from being handled by another instance.  I'm wondering if there is a better way to do this that keeps with the original pub/sub model.

    From what I've been able to read I believe this is referred to as a "shared durable subscription".  This approach relies on the subscriber ID and client ID of each instance being the same.  Does this require JMS?  I found this approach mentioned in documentation for WAS ND (not MQ, itself).

    With cloud deployment and automatic scaling by Kubernetes it seems this would be a common scenario to cover.   I'm looking for the best pattern(s) for handling client scaling when you have multiple instances of the same client subscribed to a topic -- and the desire is to NOT have each client instance process the same message.

    Thanks,
    Jim

    ------------------------------
    Jim Creasman
    ------------------------------


  • 2.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Tue September 21, 2021 06:55 PM
    Hi Jim,

    I don't use programmatic subscriptions a great deal, so I may have missed something, but I can't see a way to implement what you need with the MQSUB call. Your proposed solution of an administrative subscription putting messages onto a queue, and then having multiple applications instances share access to the queue seems to be the right approach.

    Regards,


    ------------------------------
    Neil Casey
    Senior Consultant
    Syntegrity Solutions
    Melbourne, Victoria
    IBM Champion (Cloud) 2019-21
    ------------------------------



  • 3.  RE: MQ subscriptions with multiple client instances

    Posted Thu September 30, 2021 08:43 AM
    With MQ, your only options for sharing a single application-defined subscription between multiple application instances are JMS 2.0 or AMQP. As Neil says, your approach using an admin sub and having applications consume from queues is a perfectly valid alternative if you're not using one of those APIs.

    ------------------------------
    Matthew Whitehead
    ------------------------------



  • 4.  RE: MQ subscriptions with multiple client instances

    Posted Wed March 22, 2023 09:15 AM

    I have the same scenario, but in my case the applications that pusblisher and subscriber are in NodeJS using MQTT on IBM MQ Server. 

    Am I out too? 



    ------------------------------
    Renan Vasconcelos
    ------------------------------



  • 5.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Wed March 22, 2023 10:44 AM

    Renan,

         We use the topic-to-queue pattern in our uniform clusters in this scenario.  The producer publishes to a topic.  An administrative subscription is defined at the queue manager which copies the message to a queue.  The consumers are balanced across the cluster and listening for messages on the queues.  You have to have at least as many consumers as there are queue managers in the cluster for this to work.  See my blog post here for more details.  Our clients are mostly NodeJS as well.

    Jim



    ------------------------------
    Jim Creasman
    ------------------------------



  • 6.  RE: MQ subscriptions with multiple client instances

    Posted Wed March 22, 2023 01:42 PM

    Hi Jim,

    I didn't know that NodeJs could read messages from queue in IBM MQ. It's possible? Can you to show me?

    I have also this scenario but the application that process the que  message use JMS, this case no have a problem. 



    ------------------------------
    Renan Vasconcelos
    ------------------------------



  • 7.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Thu March 23, 2023 10:08 AM

    We use the mq-mqi-nodejs package that is supported by IBM.  There are many sample programs included that show the particulars.  We provide a URL that returns the CCDT information as a connection option.  This allows the underlying MQ client / server to manage the connection to our uniform cluster.  Let me know if you have specific questions.

    With MQ 9.3 you also have the option to use rhea and AMQP when connecting to a queue.  However, this package won't work as well with uniform clusters b/c it doesn't support CCDT and connection balancing.



    ------------------------------
    Jim Creasman
    ------------------------------



  • 8.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Fri March 24, 2023 09:45 AM

    I would recommend using CCDT JSON for node; than calling a TAB file. As the node package can move the JSON and developer feels better using JSON which can be read.



    ------------------------------
    om prakash
    ------------------------------



  • 9.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Fri October 01, 2021 08:35 AM
    Edited by Francois Brandelik Fri October 01, 2021 08:35 AM
    If you are using a uniform cluster, you could also make an administrative subscription with a cluster alias. That will load balance the subscription in the cluster. You then need enough consumers to consume from each queue. This of course means that there is no message affinity.

    ------------------------------
    Francois Brandelik
    ------------------------------



  • 10.  RE: MQ subscriptions with multiple client instances

    IBM Champion
    Posted Fri March 24, 2023 09:42 AM

    The approach for Topic -> Queue is the right approach. This gaurantees the message is delivered. 

    This approach needs adminitrative/config setup to maintain a large set of Subscriptions/Queues. But with all the automation available; should be easy to manage.

    In Uniform Cluster if it is P2P; Pub/Sub is another approach. Request/Reply can be tricky in uniform cluster.

    if you have limited subscriptions with no Regex pattern, have the producer write to an alais queue, and set the alias queue type to topic; this would publish the mesage if the publisher cannot issue MQTOPIC.



    ------------------------------
    om prakash
    ------------------------------