MQ

MQ

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  IBM MQ -> AWS Kinesis

    Posted Tue November 17, 2020 02:07 AM
    Hello experts. 

    I am new to MQ. I need to build a pipeline to pull messages out of IBM MQ (in IBM cloud) into Kinesis Data Streams (in AWS). There are existing applications in IBM cloud that are putting messages in IBM MQ (in IBM cloud). The BI application is on AWS where I need to process the messages further for BI consumption.

    I have found source connectors for Kafka that can read messages from MQ, but I've not found any such readily available connectors that can do the same for Kinesis Data Streams. I am currently exploring two custom solutions:
    1. Using Apache Camel as the bridge
    2. Using PyMQI 

    Is there any other option that I should be looking at? Would you have any recommendations on using any existing connector or a custom bridge solution?

    Thanks a lot for your time and help.

    ------------------------------
    Ashish Naidu
    Principal Architect
    InfoCepts
    India
    ------------------------------


  • 2.  RE: IBM MQ -> AWS Kinesis

    Posted Tue November 17, 2020 01:43 PM
    Hello,

    There are  key differences between MQ and Streams products like Kafka.
    MQ is a enterprise messaging middleware. It is intent to enterprise point-to-point messages and publish-subscribe integration models. It enables desired features in pub/sub and point-to-point that doesn't exist or are difficult to implement in Kafka.
    Some competitors of MQ are: AMQ (Apache, RedHat), Rabbit MQ, etc. Not Kafka directly. Which doesn't mean you can't choose between both depending on what challenge you have to integrate apps.
    Streams computing is different, and Kafka is better suite to it. Kafka is a log system utilized to compute on the fly data (streams). IBM has an implementation of Kafka called IBM Event Streams. There are other services implementations of Kafka in the cloud, as well implementations of MQ and other message systems. But Kafka has a extremely difficult implementation to address pub/sub management which is very common in the Enterprise pub/sub area.
    So it is not a matter of one is better than the other.Correct is to say they are focused in different areas: Enterprise middleware messaging versus Stream Computing.

    In your case you should take a look into:

    Obs.: event Streams is based on Kafka so you can extrapolate to all Kafka products.
    Regards,

    Fernando F. Tavares
    IT Architect - Senior Middleware Integration Engineer
    Hybrid Cloud Services


    ------------------------------
    Fernando Tavares
    IT Specialist
    IBM
    Santos
    5.51397E+12
    ------------------------------

    Original Message


  • 3.  RE: IBM MQ -> AWS Kinesis

    Posted Wed November 18, 2020 02:36 AM
    Hello Ashish,

    You might also have a look to our AppConnect Toolkit.
    It's an application integration solution that allows to build flows with a no coding approach.
    It provides connector nodes for MQ and Kafka.
    You should be able to build the flow that you are looking very straightforward and quickly.
    You can deploy the flow as bar file on a AppConnect instance running on IBM Cloud.
    Some documentation:
    https://community.ibm.com/community/user/middleware/viewdocument/enhancements-in-ace-v11-for-integra?CommunityKey=77544459-9fda-40da-ae0b-fc8c76f0ce18&tab=librarydocuments

    https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools.mft.doc/bz91030_.html

    To install AppConnect toolkit:
    https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.iib.localhelp.doc/doc0004_.htm

    ------------------------------
    Pierre Richelle
    IBM Hybrid Cloud Integration Specialists
    IBM
    bruxelles
    0474681892
    ------------------------------

    Original Message