Power

 View Only

  • 1.  Debezium + Kafka for DB2 hosted on IBM i

    Posted Tue September 26, 2023 10:40 AM

    Has anyone had any luck using Debezium + Kafka to stream CDC events in DB2 databases hosted on AS400/IBMi? 

    Debezium connector for Db2

    If you have or have come across any articles on the web on how to put together a POC, can you please share them? I'm asking because, from what I've read online, there are some slight differences between DB2 on Linux and DB2 on IBMi. 



    ------------------------------
    Timothy
    ------------------------------


  • 2.  RE: Debezium + Kafka for DB2 hosted on IBM i

    IBM Champion
    Posted Wed September 27, 2023 02:09 AM
    Edited by Satid Singkorapoom Wed September 27, 2023 02:13 AM

    Dear Timothy

    The description provided by Debezium appears that it was never tested with DB2 for i.  And its product information indicates it uses something equivalent to a journal receiver object in IBM i which unfortunately is not a table that Debezium uses.  I would be surprised if it works with DB2 for i.

    Since the availability of TIMESTAMP data type and automatic timestamp in DB2 for i in release 6.1, one can write a poor-man Change Data Capture program by making use of this new timestamp column (added to the table of interest) and SQL date-time arithmetic capability.  An SQL procedure designed to run CDC once every 24 hours would be easier to write than one designed to run on-line. 

    ------------------------------
    Chance favors only the prepared mind.
    -- Louis Pasteur
    ------------------------------
    Satid S.
    ------------------------------

    Original Message


  • 3.  RE: Debezium + Kafka for DB2 hosted on IBM i

    Posted Wed September 27, 2023 09:54 AM
    Edited by Timothy Dilbert Wed September 27, 2023 09:58 AM

    Hi Satid,

    Thank you for your response. I like your idea of using a TIMESTAMP column, except that we'd also like to stream events in DB2 databases related to third-party applications. I'm concerned modifying their tables will void our warranty/support. I have the same concerns with a trigger-based approach since triggers are part of SQL transactions. 

    After posting here, I was directed to a StackOverflow article where a poster (claiming to be an IBM employee) mentioned that people should look at IBM Data Replication (click here). I have plans to explore IDR since it also supports other sources common in my client networks (e.g. MSSQL and Oracle).

    I will make sure to keep everyone posted with my findings. Unless you have experience with this and can potentially help raise any hidden caveats? 



    ------------------------------
    Timothy
    ------------------------------

    Original Message


  • 4.  RE: Debezium + Kafka for DB2 hosted on IBM i

    IBM Champion
    Posted Wed September 27, 2023 02:16 AM
      |   view attached

    I attach the article on this automatic timestamp support feature by Kent Milligan herewith. 



    ------------------------------
    Chance favors only the prepared mind.
    -- Louis Pasteur
    ------------------------------
    Satid S.
    ------------------------------

    Attachment(s)

    pdf
    DB-V6R1 Automatic Timestamp Support.pdf   131 KB 1 version
    Original Message


  • 5.  RE: Debezium + Kafka for DB2 hosted on IBM i

    Posted Fri August 02, 2024 05:25 PM

    Hi Timothy, 

    Not sure if you still are interested in using Debezium for CDC on IBM i.


    One rock solid CDC software I have experience with is Stelo (formerly StarQuest). They have very performant CDC to other DBs and cloud options. 

    Also , Debezium did enhance their plugins to support IBM i

    Reference: https://debezium.io/blog/2024/03/06/debezium-2-6-beta1-released/
    https://github.com/debezium/debezium-connector-ibmi?tab=readme-ov-file

    If you need to explore or bounce ideas for CDC options still. Ping me an email. Id be happy to assist. 

    Marius



    ------------------------------
    Marius Louis le Roux
    Owner
    MLR Consulting
    ------------------------------

    Original Message


  • 6.  RE: Debezium + Kafka for DB2 hosted on IBM i

    Posted Mon August 05, 2024 04:27 PM

    Hi Marius,

    Yes, definitely still interested. My issue is that I don't have an AS400 or experience with it to test this myself. I can do the Kubernetes and Debezium stuff, just need an environment and help configuring the DB2 side.

    Got your email from your website. Going to email you now.



    ------------------------------
    Timothy
    ------------------------------

    Original Message