Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

View Only

Expand all | Collapse all

Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

1. Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

0 Like
RAJNI HARYANI
Posted Thu January 12, 2023 03:05 AM

Reply
------------------------------
RAJNI HARYANI
------------------------------

#CloudPakforDataGroup

2. RE: Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

Like

YONG LI

Posted Mon January 16, 2023 12:41 PM

For DataStage flow, the simplest way to move from DEV to TEST to PRODUCTION is via cpdctl dsjob CLI.

You can download cpdctl tool here: https://github.com/IBM/cpdctl/releases/, please pick the one appropriate for your platform.

Read this document to understand how to use it: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=data-datastage-command-line-tools

Example to export all DataStage objects into a zip file:

cpdctl dsjob export-datastage-assets --project MyProject --file-name MyProject.zip
...
2023-01-13 19:29:49: Waiting until export finishes, Status: started
2023-01-13 19:29:50: Project export status: in_progress, total: 202, completed: 6, failed: 0, skipped: 0.
2023-01-13 19:29:51: Project export status: in_progress, total: 202, completed: 14, failed: 0, skipped: 0.
2023-01-13 19:29:53: Project export status: in_progress, total: 202, completed: 21, failed: 0, skipped: 0.
2023-01-13 19:29:54: Project export status: in_progress, total: 202, completed: 30, failed: 0, skipped: 0.
2023-01-13 19:29:56: Project export status: in_progress, total: 202, completed: 74, failed: 0, skipped: 0.
2023-01-13 19:29:57: Project export status: in_progress, total: 202, completed: 97, failed: 0, skipped: 0.
2023-01-13 19:29:59: Project export status: in_progress, total: 202, completed: 122, failed: 0, skipped: 0.
2023-01-13 19:30:00: Project export status: in_progress, total: 202, completed: 133, failed: 0, skipped: 0.
2023-01-13 19:30:01: Project export status: in_progress, total: 202, completed: 151, failed: 0, skipped: 0.
2023-01-13 19:30:03: Project export status: in_progress, total: 202, completed: 159, failed: 0, skipped: 0.
2023-01-13 19:30:06: Project export status: in_progress, total: 202, completed: 183, failed: 0, skipped: 0.
2023-01-13 19:30:09: Project export status: completed, total: 202, completed: 194, failed: 0, skipped: 0.
Objects Added:
Connections:    6
	 Yong-JDBC-Oracle-Softlayer
	 Yong-Replication-Kafka-MultiTopic
	 Yong-Replication-Oracle-ExtremEngine
	 Yong-Replication-RDS-PostgreSQL
	 Yong-kafka
	 storage_volumn
ParamSets:      1
	 PS_Test
XMLLibs:        1
	 Batter/Batter/Batter Library

Now you can import this zip file into your Test project on the same or different cluster:

cpdctl dsjob import-zip --project TestProject --file-name MyProject.zip --wait 600
...
2023-01-13 19:39:12: Waiting until import finishes, import id: b3204463-e903-4382-a0a4-2db076745c8a
2023-01-13 19:39:41: Project import status: started,  total: 200, completed: 10, failed: 0, skipped: 0.
2023-01-13 19:40:43: Project import status: started,  total: 200, completed: 15, failed: 0, skipped: 0.
2023-01-13 19:41:13: Project import status: started,  total: 200, completed: 60, failed: 0, skipped: 0.
2023-01-13 19:41:43: Project import status: started,  total: 200, completed: 101, failed: 0, skipped: 1.
2023-01-13 19:42:13: Project import status: started,  total: 200, completed: 115, failed: 0, skipped: 1.
2023-01-13 19:42:43: Project import status: started,  total: 200, completed: 131, failed: 0, skipped: 1.
2023-01-13 19:43:43: Project import status: started,  total: 200, completed: 134, failed: 0, skipped: 1.
2023-01-13 19:45:13: Project import status: started,  total: 200, completed: 148, failed: 0, skipped: 1.
2023-01-13 19:45:43: Project import status: started,  total: 200, completed: 157, failed: 0, skipped: 1.
2023-01-13 19:47:13: Project import status: started,  total: 200, completed: 162, failed: 0, skipped: 1.
2023-01-13 19:48:13: Project import status: started,  total: 200, completed: 171, failed: 0, skipped: 1.
2023-01-13 19:48:43: Project import status: started,  total: 200, completed: 199, failed: 1, skipped: 1.
2023-01-13 19:49:13: Project import status: completed,  total: 200, completed: 200, failed: 1, skipped: 1.

Status code = 0

You can also choose to move flow by flow, read the documentation above to find out details.

Hope this helps!

------------------------------
YONG LI
------------------------------

Original Message

3. RE: Any CI/CD concept in IBM Cloud pak means if I have created a datastage flow in development environment how to send this flow to TEST env automatically?

Like

YONG LI

Posted Mon January 16, 2023 12:41 PM

You can use cpdctl dsjob to export and import project or individual DataStage flows.

cpdctl can be downloaded from here: https://github.com/IBM/cpdctl/releases/
Read documentation here: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.6.x?topic=data-datastage-command-line-tools

Example to export a project:
cpdctl dsjob export-datastage-assets --project DevProject --file-name DevProject.zip
...
2023-01-13 19:29:49: Waiting until export finishes, Status: started
2023-01-13 19:29:50: Project export status: in_progress, total: 202, completed: 6, failed: 0, skipped: 0.
2023-01-13 19:29:51: Project export status: in_progress, total: 202, completed: 14, failed: 0, skipped: 0.
2023-01-13 19:29:53: Project export status: in_progress, total: 202, completed: 21, failed: 0, skipped: 0.
2023-01-13 19:29:54: Project export status: in_progress, total: 202, completed: 30, failed: 0, skipped: 0.
2023-01-13 19:29:56: Project export status: in_progress, total: 202, completed: 74, failed: 0, skipped: 0.
2023-01-13 19:29:57: Project export status: in_progress, total: 202, completed: 97, failed: 0, skipped: 0.
2023-01-13 19:29:59: Project export status: in_progress, total: 202, completed: 122, failed: 0, skipped: 0.
2023-01-13 19:30:00: Project export status: in_progress, total: 202, completed: 133, failed: 0, skipped: 0.
2023-01-13 19:30:01: Project export status: in_progress, total: 202, completed: 151, failed: 0, skipped: 0.
2023-01-13 19:30:03: Project export status: in_progress, total: 202, completed: 159, failed: 0, skipped: 0.
2023-01-13 19:30:06: Project export status: in_progress, total: 202, completed: 183, failed: 0, skipped: 0.
2023-01-13 19:30:09: Project export status: completed, total: 202, completed: 194, failed: 0, skipped: 0.
Objects Added:
Connections:    6
    My-JDBC-Oracle-Softlayer
    My-Replication-Kafka-MultiTopic
    My-Replication-Oracle-ExtremEngine
    My-Replication-RDS-PostgreSQL
    My-kafka
    storage_volumn
ParamSets:      1
    PS_Test
XMLLibs:        1
    Batter/Batter/Batter Library

Import this zip into the Test project on the same or different cluster:
cpdctl dsjob import-zip --project TestProject --file-name DevProject.zip --wait 600
...
2023-01-13 19:39:12: Waiting until import finishes, import id: b3204463-e903-4382-a0a4-2db076745c8a
2023-01-13 19:39:41: Project import status: started,  total: 200, completed: 10, failed: 0, skipped: 0.
2023-01-13 19:40:43: Project import status: started,  total: 200, completed: 15, failed: 0, skipped: 0.
2023-01-13 19:41:13: Project import status: started,  total: 200, completed: 60, failed: 0, skipped: 0.
2023-01-13 19:41:43: Project import status: started,  total: 200, completed: 101, failed: 0, skipped: 1.
2023-01-13 19:42:13: Project import status: started,  total: 200, completed: 115, failed: 0, skipped: 1.
2023-01-13 19:42:43: Project import status: started,  total: 200, completed: 131, failed: 0, skipped: 1.
2023-01-13 19:43:43: Project import status: started,  total: 200, completed: 134, failed: 0, skipped: 1.
2023-01-13 19:45:13: Project import status: started,  total: 200, completed: 148, failed: 0, skipped: 1.
2023-01-13 19:45:43: Project import status: started,  total: 200, completed: 157, failed: 0, skipped: 1.
2023-01-13 19:47:13: Project import status: started,  total: 200, completed: 162, failed: 0, skipped: 1.
2023-01-13 19:48:13: Project import status: started,  total: 200, completed: 171, failed: 0, skipped: 1.
2023-01-13 19:48:43: Project import status: started,  total: 200, completed: 199, failed: 1, skipped: 1.
2023-01-13 19:49:13: Project import status: completed,  total: 200, completed: 200, failed: 1, skipped: 1.

Status code = 0

You can also export individual DataStage flow as a zip file from your Dev project and import into into your Test project.

------------------------------
YONG LI
------------------------------

Original Message

IBM Data Lifecycle - Integration and Governance

Connect with experts and peers to elevate technical expertise, solve problems and share insights.

Cloud Pak for Data