DataStage - Data Integration

 View Only
Expand all | Collapse all

How to insert objects into MONGODB with DataStage

  • 1.  How to insert objects into MONGODB with DataStage

    Posted Thu May 20, 2021 08:26 PM
    A customer wants to use the ODBC/JDBC connector to LOAD objects into MONGODB with DataStage 11.7, Does anyone has a guide of the capabilities inserting objects into MONGODB:ips-question: is it possible :question_mark_double:
    Te image shows and ID inserted as a String (Second tree) with DataStage, but they need to insert an object like the first ID Tree inserted.
    Any ideas??? any documentation or guide?
    The image shows and ID inserted with DataStage as a String (Second tree), but the requirement is to insert an object like the first ID Tree.


    ------------------------------
    VICTOR JIMENEZ SANCHEZ
    DataStage Consultant
    IBM
    ------------------------------


  • 2.  RE: How to insert objects into MONGODB with DataStage

    Posted Tue September 21, 2021 02:32 AM
    Hi

    These are the link and indicate what is not supported if you are leveraging on the ODBC Connector and the ODBC Driver provided - https://www.ibm.com/docs/en/iis/11.7?topic=connector-connecting-mongodb-database

    The other alternative is to leverage Hierarchical Stage and REST API.

    ------------------------------
    HOW MING YONG
    ------------------------------



  • 3.  RE: How to insert objects into MONGODB with DataStage

    Posted Wed September 22, 2021 08:10 AM
    The truth is that the JDBC/ODBC Connector for MongoDB is VERY limited. Also you will require another process running on the DS Engine server to map your objects/arrays. Using the hierarchical stage to "unwind" your arrays into flat records is a nightmare (in my opinion the worst stage available)... 

    We ended up creating our own connector by using the Java Connector Stage with the MongoDB Java driver. We are streaming huge the JSON files with the Jackson API and loading them in bulk in MongoDB. Streaming huge files with the hierarchical stage is the last thing you want to do. 

    Don't get me wrong DS is a great ETL tool but when it comes to JSON and a Document store like MongoDB it is very painful and inefficient to use.

    ------------------------------
    Emmanuel Côté
    ------------------------------