MQ

Expand all | Collapse all

Migrate from MQ Appliance M2001B to IBM MQ Cloud.

  • 1.  Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 05:41 AM
    Hi,
    At this moment we run MQ Appliance M2001B, we run this since June 2016. In the next 5 years we move to a cloud based environment in AWS. Most of our  cloud services are located in Amazons AWS Frankfurt. Because our JMS based queueing mechanisme is solid and reliable and high available we want to keep using this messaging principle also in the cloud.

    There a couple of solutions  IBM MQ is delivering for the cloud, the solution where this post is about is: 

    IBM MQ Cloud (MQ as a service in the cloud).

    If we move form having  MQ Appliances high available configured to IBM MQ Cloud I don't want the producers, consumers configuration and API's less as possible.

    Questions:
    1. Can I keep the Java JMS based architecture ?
    2. Can I use the MQ AllClient 9.x.x.jar to connect the JMS based MQ Client to the IBM MQ Cloud instance qmanager.
    3. How is High Availablitiy aranged on IBM MQ Cloud ?
    4. Can I use MQ Explorer on my local system for example via port 1414 to connect to the Queuemanager in the cloud ?
    5. Is it possible to use LDAP(Active directory) to use as authentication mechanism on IBM MQ Cloud.
    Thanks

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------


  • 2.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 06:42 AM
    Hello.

    In short:

    1. Yes
    2. Yes
    3. See https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-mqoc_ha
    4. Yes
    5. No - we only authenticate via IBMid / IAM. For standalone MQ you can configure the authentication to be done using LDAP instead of using OS users. However MQoC uses OS with custom PAM module that calls out to qmgr-auth and ultimately IAM.

    Thanks - please feel free to reach out to me directly if you have any more questions I can help with.

    ------------------------------
    Jack Boad - Offering Manager - IBM MQoC
    jack.boad@uk.ibm.com
    ------------------------------



  • 3.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 07:59 AM
    Edited by Bernard Pittens Tue July 14, 2020 08:00 AM
    Hi Jack,
    Thanks, I have read the link to the HA section at point 3 and the Cold disaster recovery. Let me check if I understand this right:
    Scenario:
    We use a Qmanager in Frankfurt(QMF) and a Qmanager in Londen(QML), Qmanagers are HA configured like stated in the link.
    A producer puts a message on QMF, before the consumer can get the message the Frankfurt datacenter crashes and the K8 cluster and storage is gone in Frankfurt. Londen(QML) is still available and running.
    Is the message on QMF lost ?

    In our current configuration we use the MQ Appliance in HA config(two datacenters) every message is replicated before it is commited and so it is never lost.

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------



  • 4.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 08:45 AM
    Hi,

    I think you are correct, yes. In this setup there is no link between QML and QMF afaik. It is up to the consumers/producers to auto connect to the mirrored QM and resume work.

    Normally if you use persistent messages the message will become available when the queue manager comes back up. If the data center is toast and we need to restore the queue manager to another AZ messages are lost.

    Thanks

    ------------------------------
    Jack Boad
    Offering Manager - IBM MQoC
    ------------------------------



  • 5.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 09:16 AM
    Thanks Jack,
    This means for Sligro that this is not a  HA solution on data.
    That is bad news, because we are not allowed to lose messages.

    I think IBM has some work to do: create or extend the IBM MQ Cloud system with replicated storage over multiple Regions or Availability zones to guarantee messages are never lost.
    Kind regards
    Bernard

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------



  • 6.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 10:23 AM
    Hi,

    Thanks for your feedback - we will take it on board. We have heard similar from other customers and we are working to see when we can fit a good HA story into our roadmap.

    Thanks

    ------------------------------
    Jack Boad
    Offering Manager - IBM MQoC
    ------------------------------



  • 7.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 06:53 AM

    A little more on your Q.4

    Not only can you continue to use MQ Explorer, but you can continue to use any client connected MQ administration tool. In essence, to use a tool with IBM MQ on IBM Cloud, you need to be able use a client connection, and to specify a user ID and password on the connect call.

    I mention this because we have a few :-)

    Please see Use MQGem tools with IBM MQ on IBM Cloud

    Cheers,
    Morag



    ------------------------------
    Morag Hughson
    MQ Technical Education Specialist
    MQGem Software Limited
    ------------------------------



  • 8.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 08:13 AM
    Hi Morag,
    Thats great thanks, thats a big plus. The challenge is the HA configuration like  I understand. See also answer an question to Jack.
    Kind regards
    Bernard

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------



  • 9.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 01:21 PM
    "create or extend the IBM MQ Cloud system with replicated storage over multiple Regions or Availability zones to guarantee messages are never lost."

    Are you willing to incur the performance hit of waiting for the data of every MQ API call to be replicated and committed hundreds or thousands of miles away? That is the only way you can "guarantee" a message won't be lost. As soon as you introduce asynchronous data replication to the alternate data center, you introduce the potential for losing or duplicating an MQ message. (The duplication risk is when the successful MQGET call was not yet replicated/committed at moment of disaster and the already once consumed message is there again in DR)

    I think you are asking for more availability than your MQ Appliances are offering. The Active/Active pair of appliances have to be collocated for synchronous data replication to guarantee no data loss. That's your H.A. and that is what the multiple worker nodes in the Kubernetes solution in MQoC mimics. How far apart are your 2 datacenters that each have one of the 2 appliances yet still allows for synchronous replication? Close enough that some may argue that is only H.A and not "real" DR, but its debatable.

    For out of region DR, the 3rd appliance for DR is getting replicated to asynchronously, and that means potential message loss or duplication. If your apps can currently tolerate missing or duplicate messages for DR, they should be able to tolerate missing messages in the DR scenario in the MQoC offering. If they can't, they are already at risk and just don't know it.

    I guess a question to ask is one Region or Availability zone in a Cloud offering more or less resilient than a private company's 2 datacenters that are close enough for synchronous replication for H.A and some degree of DR, maybe enough DR. It would be nice to have HA and "DR" for local mini disasters with the entire solution close enough for synchronous replication and then remote DR offerings that offer out of region recover ability but at the cost of no stateful data like messages being replicated (or if replicated, asynchronously and not guaranteed in any way).

    ------------------------------
    Peter Potkay
    ------------------------------



  • 10.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Tue July 14, 2020 03:04 PM
    Peter,

    Thanks for your response because you have some very interesting points,

    Let me respond  on your message:
    "I think you are asking for more availability than your MQ Appliances are offering. The Active/Active pair of appliances have to be collocated for synchronous data replication to guarantee no data loss. That's your H.A. and that is what the multiple worker nodes in the Kubernetes solution in MQoC mimics. How far apart are your 2 datacenters that each have one of the 2 appliances yet still allows for synchronous replication? Close enough that some may argue that is only H.A and not "real" DR, but its debatable."

    Geographical distance between the datacenters at Sligro Food group is  1,2, KM. And yes it is a synchronous replication where we can rely on. We had crashes of one MQ Appliance before and the other Appliance continued without any problem, they are configured as fail over: One Active and one passive device.  We also have tested the option for DR with the MQ Appliances but we did not choose for this because its a-synchronous and messages might be lossed.

    Ok, then about the Kubernetes(K8) solution because maybe I understand this wrong, lets go into the details:
    The workers in IBM MQ Cloud:
    The K8 solution uses workers, they are replicated and if one goes down another takes over I see in the docs. But are there more as one worker running at the same time, so if a worker goes down my MQ client apps does not notice this ?  
    Then the storage for persistent messaging:
    I guess the system uses a PVC (Persistant volume claim) ??  Somewhere in the AZ there is a  server where the storage is located. I have not enough knowledge about K8 what the possibilitys are in replicating this storage in the same AZ. 
    Is this storage location replicated, and how is this done, is there any guarantee of message persistence by IBM ? 

    Thats important in my opinion.

    I have noticed in the search for  a 100% fail save HA queueing solution that indeed it takes time to replicate messages in a cloud with multi AZ in AWS. So indeed it is maybe not possbile to have a synchronous data replication on multi AZ and the challenge on multi region is even bigger.  IBM Cloud MQ is using different regions and not different AZ's in the same region, which is also a performance hit.
    Yes and when the data replication is a-synchronous then it is indeed the question or we gain any advantage like you describe.

    We tested also AmazonMQ. Amazon MQ uses synchronous data replication on multi AZ for example but it gives indeed a performance hit we noticed and is 5 times slower as IBM MQ in message replication. See also this link at my personal  Stack Overflow account:
    https://stackoverflow.com/questions/60853007/amazon-mq-activemq-bad-performance-on-large-messages/

    Thanks.
    Kind regards
    Bernard

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------



  • 11.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Wed July 29, 2020 11:08 AM
    Hi Bernard

    Have you considered using RDQM as a possible option?  My  client is running MQ in AWS and using this as a solution for HA.  It does require close links between servers so works well in an AWS region -  in our case just in London so we have an MQ server in each of the 3 availability zones.  When we have failed over it has taken a matter of seconds and we haven't seen any data loss - all our queues are persistent.

    You also have the option to combine the RDQM HA option with RDQM DR  (from 9.2 LTS I believe).

    Regards

    John


    ------------------------------
    John Hirst
    ------------------------------



  • 12.  RE: Migrate from MQ Appliance M2001B to IBM MQ Cloud.

    Posted Wed July 29, 2020 12:07 PM
    Hi John,
    Yes RDQM look like a very interesting option, I am going to try this in a POC.
    Thanks
    Kind regards

    ------------------------------
    Bernard Pittens
    Integration Engeneer
    Sligro Foodgroup B.V.
    Veghel
    ------------------------------