Cloud Pak for Data

 View Only
Expand all | Collapse all

Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

  • 1.  Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

    Posted Thu January 12, 2023 03:05 AM


    ------------------------------
    RAJNI HARYANI
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

    Posted Mon January 16, 2023 12:41 PM
    For DataStage flow, the simplest way to move from DEV to TEST to PRODUCTION is via cpdctl dsjob CLI.

    You can download cpdctl tool here: https://github.com/IBM/cpdctl/releases/, please pick the one appropriate for your platform.

    Read this document to understand how to use it: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=data-datastage-command-line-tools

    Example to export all DataStage objects into a zip file:
    cpdctl dsjob export-datastage-assets --project MyProject --file-name MyProject.zip
    ...
    2023-01-13 19:29:49: Waiting until export finishes, Status: started
    2023-01-13 19:29:50: Project export status: in_progress, total: 202, completed: 6, failed: 0, skipped: 0.
    2023-01-13 19:29:51: Project export status: in_progress, total: 202, completed: 14, failed: 0, skipped: 0.
    2023-01-13 19:29:53: Project export status: in_progress, total: 202, completed: 21, failed: 0, skipped: 0.
    2023-01-13 19:29:54: Project export status: in_progress, total: 202, completed: 30, failed: 0, skipped: 0.
    2023-01-13 19:29:56: Project export status: in_progress, total: 202, completed: 74, failed: 0, skipped: 0.
    2023-01-13 19:29:57: Project export status: in_progress, total: 202, completed: 97, failed: 0, skipped: 0.
    2023-01-13 19:29:59: Project export status: in_progress, total: 202, completed: 122, failed: 0, skipped: 0.
    2023-01-13 19:30:00: Project export status: in_progress, total: 202, completed: 133, failed: 0, skipped: 0.
    2023-01-13 19:30:01: Project export status: in_progress, total: 202, completed: 151, failed: 0, skipped: 0.
    2023-01-13 19:30:03: Project export status: in_progress, total: 202, completed: 159, failed: 0, skipped: 0.
    2023-01-13 19:30:06: Project export status: in_progress, total: 202, completed: 183, failed: 0, skipped: 0.
    2023-01-13 19:30:09: Project export status: completed, total: 202, completed: 194, failed: 0, skipped: 0.
    Objects Added:
    Connections:    6
    	 Yong-JDBC-Oracle-Softlayer
    	 Yong-Replication-Kafka-MultiTopic
    	 Yong-Replication-Oracle-ExtremEngine
    	 Yong-Replication-RDS-PostgreSQL
    	 Yong-kafka
    	 storage_volumn
    ParamSets:      1
    	 PS_Test
    XMLLibs:        1
    	 Batter/Batter/Batter Library
    ​
    Now you can import this zip file into your Test project on the same or different cluster:

    cpdctl dsjob import-zip --project TestProject --file-name MyProject.zip --wait 600
    ...
    2023-01-13 19:39:12: Waiting until import finishes, import id: b3204463-e903-4382-a0a4-2db076745c8a
    2023-01-13 19:39:41: Project import status: started,  total: 200, completed: 10, failed: 0, skipped: 0.
    2023-01-13 19:40:43: Project import status: started,  total: 200, completed: 15, failed: 0, skipped: 0.
    2023-01-13 19:41:13: Project import status: started,  total: 200, completed: 60, failed: 0, skipped: 0.
    2023-01-13 19:41:43: Project import status: started,  total: 200, completed: 101, failed: 0, skipped: 1.
    2023-01-13 19:42:13: Project import status: started,  total: 200, completed: 115, failed: 0, skipped: 1.
    2023-01-13 19:42:43: Project import status: started,  total: 200, completed: 131, failed: 0, skipped: 1.
    2023-01-13 19:43:43: Project import status: started,  total: 200, completed: 134, failed: 0, skipped: 1.
    2023-01-13 19:45:13: Project import status: started,  total: 200, completed: 148, failed: 0, skipped: 1.
    2023-01-13 19:45:43: Project import status: started,  total: 200, completed: 157, failed: 0, skipped: 1.
    2023-01-13 19:47:13: Project import status: started,  total: 200, completed: 162, failed: 0, skipped: 1.
    2023-01-13 19:48:13: Project import status: started,  total: 200, completed: 171, failed: 0, skipped: 1.
    2023-01-13 19:48:43: Project import status: started,  total: 200, completed: 199, failed: 1, skipped: 1.
    2023-01-13 19:49:13: Project import status: completed,  total: 200, completed: 200, failed: 1, skipped: 1.
    
    Status code = 0
    ​


    You can also choose to move flow by flow, read the documentation above to find out details.

    Hope this helps!



    ------------------------------
    YONG LI
    ------------------------------



  • 3.  RE: Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

    Posted Mon January 16, 2023 12:41 PM
    You can use cpdctl dsjob to export and import project or individual DataStage flows.

    cpdctl can be downloaded from here: https://github.com/IBM/cpdctl/releases/
    Read documentation here: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=data-datastage-command-line-tools

    Example to export a project:
    cpdctl dsjob export-datastage-assets --project DevProject --file-name DevProject.zip
    ...
    2023-01-13 19:29:49: Waiting until export finishes, Status: started
    2023-01-13 19:29:50: Project export status: in_progress, total: 202, completed: 6, failed: 0, skipped: 0.
    2023-01-13 19:29:51: Project export status: in_progress, total: 202, completed: 14, failed: 0, skipped: 0.
    2023-01-13 19:29:53: Project export status: in_progress, total: 202, completed: 21, failed: 0, skipped: 0.
    2023-01-13 19:29:54: Project export status: in_progress, total: 202, completed: 30, failed: 0, skipped: 0.
    2023-01-13 19:29:56: Project export status: in_progress, total: 202, completed: 74, failed: 0, skipped: 0.
    2023-01-13 19:29:57: Project export status: in_progress, total: 202, completed: 97, failed: 0, skipped: 0.
    2023-01-13 19:29:59: Project export status: in_progress, total: 202, completed: 122, failed: 0, skipped: 0.
    2023-01-13 19:30:00: Project export status: in_progress, total: 202, completed: 133, failed: 0, skipped: 0.
    2023-01-13 19:30:01: Project export status: in_progress, total: 202, completed: 151, failed: 0, skipped: 0.
    2023-01-13 19:30:03: Project export status: in_progress, total: 202, completed: 159, failed: 0, skipped: 0.
    2023-01-13 19:30:06: Project export status: in_progress, total: 202, completed: 183, failed: 0, skipped: 0.
    2023-01-13 19:30:09: Project export status: completed, total: 202, completed: 194, failed: 0, skipped: 0.
    Objects Added:
    Connections: 6
    My-JDBC-Oracle-Softlayer
    My-Replication-Kafka-MultiTopic
    My-Replication-Oracle-ExtremEngine
    My-Replication-RDS-PostgreSQL
    My-kafka
    storage_volumn
    ParamSets: 1
    PS_Test
    XMLLibs: 1
    Batter/Batter/Batter Library

    Import this zip into the Test project on the same or different cluster:
    cpdctl dsjob import-zip --project TestProject --file-name DevProject.zip --wait 600
    ...
    2023-01-13 19:39:12: Waiting until import finishes, import id: b3204463-e903-4382-a0a4-2db076745c8a
    2023-01-13 19:39:41: Project import status: started, total: 200, completed: 10, failed: 0, skipped: 0.
    2023-01-13 19:40:43: Project import status: started, total: 200, completed: 15, failed: 0, skipped: 0.
    2023-01-13 19:41:13: Project import status: started, total: 200, completed: 60, failed: 0, skipped: 0.
    2023-01-13 19:41:43: Project import status: started, total: 200, completed: 101, failed: 0, skipped: 1.
    2023-01-13 19:42:13: Project import status: started, total: 200, completed: 115, failed: 0, skipped: 1.
    2023-01-13 19:42:43: Project import status: started, total: 200, completed: 131, failed: 0, skipped: 1.
    2023-01-13 19:43:43: Project import status: started, total: 200, completed: 134, failed: 0, skipped: 1.
    2023-01-13 19:45:13: Project import status: started, total: 200, completed: 148, failed: 0, skipped: 1.
    2023-01-13 19:45:43: Project import status: started, total: 200, completed: 157, failed: 0, skipped: 1.
    2023-01-13 19:47:13: Project import status: started, total: 200, completed: 162, failed: 0, skipped: 1.
    2023-01-13 19:48:13: Project import status: started, total: 200, completed: 171, failed: 0, skipped: 1.
    2023-01-13 19:48:43: Project import status: started, total: 200, completed: 199, failed: 1, skipped: 1.
    2023-01-13 19:49:13: Project import status: completed, total: 200, completed: 200, failed: 1, skipped: 1.

    Status code = 0

    You can also export individual DataStage flow as a zip file from your Dev project and import into into your Test project.


    ------------------------------
    YONG LI
    ------------------------------