MQ

  • 1.  Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Wed August 18, 2021 03:56 PM
    Hi All,

    As we have a new feature for streaming queues in V 9.2.3. The document does mention some of the sceanrios where streaming queues can be used . It also mention the restrictions for using streaming queues as mentioned in this link .

    few restritions to highlight here for my question:
    • Defining a chain of queues streaming to each other, for example, Q1->Q2, Q2->Q3, Q3->Q4
    • Defining STREAMQ on a queue configured with USAGE(XMITQ)
    So, as per documentation, these cases are not supported though i think, these are technically feasible. My question is : why can't we use streaming queues for message availability across queue managers / nodes , this may insure message availability across different queue manager in HA scenario & almost 0 RPO in case of DR.

    App -> Q1 (QMGR1)-----RQ1(StreamQ & RemoteQ ) ------xmitQ -------------------------------------Q1(QMGR2)
                 
    I  undersstand , we have other  solutions for replicating messages across different queue manager nodes in HA & DR scenario ( RDQM ), but not all the queues in a queue manager are critical & if we can achive replication of message / duplicaion of message across queue manager nodes using streaming queue, that would add a granual level replication feature. ( though licensing & some performance overhead may come )

    Thanks,
    Sumit

    ------------------------------
    sumit sinha
    ------------------------------


  • 2.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Thu August 19, 2021 02:47 AM

    Hi Sumit,

    The configuration you propose is supported, as you are simply streaming from Q1 to RQ1 on QMGR1.

    The rules you highlight would mean that you shouldn't also set up streaming on Q1 on QMGR2 as that would create a chain of connected streams. And the second one prevents you from enabling a stream queue on QMGR1's xmitq instead of Q1 itself.

    We'll see if we can improve our documentation to make this clearer.

    So if you are simply looking to duplicate the messages flowing through Q1@QMGR1 so that they also arrive at Q1@QMGR2 then you definitely can.

    Depending on your needs, you could see this as an HA or DR mechanism, but it is very different to how MQ natively handles HA and DR. The main difference from an application's point of view is that only the production of the messages is being replicated with streaming queues, not the consumption of those messages, unlike something like RDQM which replicates all state changes, keeping multiple representations of the same queue in step.

    So with streaming queues you'd need your own mechanism in place to be consuming messages from Q1@QMGR2 to ensure that the queue did not over fill, yet still holds enough messages to provide you benefit in the case of a fail over from QMGR1. This would obviously expose you to possibly processing the same message multiple times in the event of a switch from QMGR1 to QMGR2, but maybe you already have ideas on how to handle that within the application.

    Thanks,
    David



    ------------------------------
    David Ware
    STSM, IBM MQ Chief Architect
    Hursley, UK
    ------------------------------



  • 3.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Thu August 26, 2021 06:51 AM

    Thanks David. 

    "The rules you highlight would mean that you shouldn't also set up streaming on Q1 on QMGR2 as that would create a chain of connected streams. And the second one prevents you from enabling a stream queue on QMGR1's xmitq instead of Q1 itself."

    I understand, technically i can but i shouldn't create a chain of connected streams & neither its supported.

    can i ask one more question : any specifc reason for launching this feature when i could achieve the same thing ( replicating messages )with MQ pub-sub.

    Thanks,

    Sumit



    ------------------------------
    sumit sinha
    ------------------------------



  • 4.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Thu August 26, 2021 07:18 AM

    Hi Sumit,

    Just to clarify, your topology is definitely supported.

    App -> Q1 (QMGR1)-----RQ1(StreamQ & RemoteQ ) ------xmitQ -------------------------------------Q1(QMGR2)

    This is because you're simply streaming from Q1 to RQ1. The downstream xmitq and target queue are not a problem, MQ handles all that. We're fixing the documentation to make that clearer.

    Pub/sub is another way of getting similar behaviour, however it has a couple of aspects that can cause problems for some scenarios:
    1. It often requires you to make more extensive configuration changes which are exposed out to the application, which isn't always simple to achieve. For example you may need to delete the original queue and replace it with an alias to a topic, or reconfigure the consumer to consume the messages from a different queue.
    2. The meta data of a message changes when using pub/sub instead of a queue. For example the message IDs will change between the producer and the consumer. This can cause problems with the original application

    The streaming queue feature is designed so that it doesn't change anything for the original producing and consuming applications on the original queue, so that you can enable this in the knowledge that they'll continue to run exactly as they did before.




    ------------------------------
    David Ware
    STSM, IBM MQ Chief Architect
    Hursley, UK
    ------------------------------



  • 5.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Fri August 27, 2021 02:57 AM
    Hi Sumit,

    the issue is that the IBM MQ implementation of pub/sub uses some of the fields in the MQMD (including the message id and correlation id).

    If you put a message to a queue and get back the message id, the message id returned in the response to the MQPUT is the same as the message id received by the application that performs the MQGET.

    If you do the MQPUT to a topic alias the message id received by the application performing the MQGET will be different to the one seen in the MQPUT response. The request/response convention is to use the message id of the request in the correl id of the response. If an application follows this convention and the message was published and replicated, then the sending application can't find the correct response message, because the correl id is wrong.

    The stream queue capability resolves this problem, because both messages (the original and the streamq copy) have the same message id. Of course, if your streamq destination is a topic alias, the message id there will be wrong, but the original applications will keep working. If you changed a working application from using a queue to using a topic alias and delivering the request message via an administrative subscription, the applications would not deliver response messages correctly.

    This was identified several versions ago and an RFE requesting enhancements to pub/sub was raised. Stream Queues address the requirement, but in a different way.

    So if you want to replicate the bodies of messages, but aren't using request/response processing, then go ahead and use pub/sub if you would prefer to. If you need to capture your request messages then Stream Queues provide a solution.

    Regards,

    ------------------------------
    Neil Casey
    Senior Consultant
    Syntegrity Solutions
    Melbourne, Victoria
    IBM Champion (Cloud) 2019-21
    ------------------------------



  • 6.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Fri August 27, 2021 07:07 AM

    Thanks Neil for the detail explanation. :) 

    Thanks,

    Sumit



    ------------------------------
    sumit sinha
    ------------------------------



  • 7.  RE: Why streaming queues can't be use for HA scenario specially for zero loss of message when the queue manager node is down

    Posted Thu August 19, 2021 01:25 PM
    Hi Sumit,

    If you want a solution that fits your description then you will need to look at a 3rd party product called MQ Message Replication.

    Regards,
    Roger

    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London ON
    226-980-7307
    ------------------------------