Data Integration

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

Dear Ralf, thank you for you response and explanations.

Your instructions are correct and we know them. Maybe I wasn't clear enough with my question.

Our client run Datastage Jobs by using Run Bash Script activities in Watson Pipelines Jobs.

To do this, include CPDCTL DSJOB RUN commands within the Run Bash Script activities. this works fine, no problem

The problem is to verify the Datastage Job finish status given that the CEL functions in expressions builders used in follows links apply to the activity status, not the Datastage Job finish status. (or at least that's what we were able to verify, where Datastage Jobs fails but the run bash activity that contains it does not fail, then the results status return ok).

Thanks in advance.

Javier.

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

Hi Javier,

so why do you just don't use a "Run DataStage/Pipeline job" activity to run your jobs in the pipeline? Is it because you want dynamically address the Jobname? If you don't use the "Run DataStage/Pipeline job" activity, then you would have to wrap the CPDCTL DSJOB RUN command into a skript that handles everything, I did this years ago in legacy DataStage with dsjob run, first checking the state of an DataStage job, resetting it if required, then run it and in the end again checking the state of the last run to check if it has been successfull.

I had a quik look at this for CP4D/CPDCTL

Command reference

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.5.0.0.md

To Run a job
cpdctl dsjob run -p PROJECT_NAME -n JOBNAME
This should always give you RC=0 if you are able run the job (e.g. you are able the connect to the prject and it exists) and will print a Status code to STDOUT which you can then parse.

In addition, you can run e.g.

cpdctl dsjob list-job-status -p PROJECT_NAME -n JOBNAME

afterwards, which will give you the Run Status of your Job.

I hope this helps.

Dear Ralf, thank you for you response and explanations.

Your instructions are correct and we know them. Maybe I wasn't clear enough with my question.

Our client run Datastage Jobs by using Run Bash Script activities in Watson Pipelines Jobs.

To do this, include CPDCTL DSJOB RUN commands within the Run Bash Script activities. this works fine, no problem

The problem is to verify the Datastage Job finish status given that the CEL functions in expressions builders used in follows links apply to the activity status, not the Datastage Job finish status. (or at least that's what we were able to verify, where Datastage Jobs fails but the run bash activity that contains it does not fail, then the results status return ok).

Thanks in advance.

Javier.

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

You can run with -wait that will gives you a return code.

• 0: successfully completed
• 1: completed with warnings
• 3: completed with error
• 4: failed
• 5: canceled
• -1: other

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.4.8.3.md#jobs

Hi Javier,

so why do you just don't use a "Run DataStage/Pipeline job" activity to run your jobs in the pipeline? Is it because you want dynamically address the Jobname? If you don't use the "Run DataStage/Pipeline job" activity, then you would have to wrap the CPDCTL DSJOB RUN command into a skript that handles everything, I did this years ago in legacy DataStage with dsjob run, first checking the state of an DataStage job, resetting it if required, then run it and in the end again checking the state of the last run to check if it has been successfull.

I had a quik look at this for CP4D/CPDCTL

Command reference

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.5.0.0.md

To Run a job
cpdctl dsjob run -p PROJECT_NAME -n JOBNAME
This should always give you RC=0 if you are able run the job (e.g. you are able the connect to the prject and it exists) and will print a Status code to STDOUT which you can then parse.

In addition, you can run e.g.

cpdctl dsjob list-job-status -p PROJECT_NAME -n JOBNAME

afterwards, which will give you the Run Status of your Job.

I hope this helps.

Dear Ralf, thank you for you response and explanations.

Your instructions are correct and we know them. Maybe I wasn't clear enough with my question.

Our client run Datastage Jobs by using Run Bash Script activities in Watson Pipelines Jobs.

To do this, include CPDCTL DSJOB RUN commands within the Run Bash Script activities. this works fine, no problem

The problem is to verify the Datastage Job finish status given that the CEL functions in expressions builders used in follows links apply to the activity status, not the Datastage Job finish status. (or at least that's what we were able to verify, where Datastage Jobs fails but the run bash activity that contains it does not fail, then the results status return ok).

Thanks in advance.

Javier.

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

Thanks Thomas,

you are correct

cpdctl dsjob run -w -1 -p PROJECT_NAME -n JOBNAME

Will wait (-1 wait for an infinite amount of time) for the job to finish and return the Status code as RC.

You can run with -wait that will gives you a return code.

• 0: successfully completed
• 1: completed with warnings
• 3: completed with error
• 4: failed
• 5: canceled
• -1: other

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.4.8.3.md#jobs

Hi Javier,

so why do you just don't use a "Run DataStage/Pipeline job" activity to run your jobs in the pipeline? Is it because you want dynamically address the Jobname? If you don't use the "Run DataStage/Pipeline job" activity, then you would have to wrap the CPDCTL DSJOB RUN command into a skript that handles everything, I did this years ago in legacy DataStage with dsjob run, first checking the state of an DataStage job, resetting it if required, then run it and in the end again checking the state of the last run to check if it has been successfull.

I had a quik look at this for CP4D/CPDCTL

Command reference

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.5.0.0.md

To Run a job
cpdctl dsjob run -p PROJECT_NAME -n JOBNAME
This should always give you RC=0 if you are able run the job (e.g. you are able the connect to the prject and it exists) and will print a Status code to STDOUT which you can then parse.

In addition, you can run e.g.

cpdctl dsjob list-job-status -p PROJECT_NAME -n JOBNAME

afterwards, which will give you the Run Status of your Job.

I hope this helps.

Dear Ralf, thank you for you response and explanations.

Your instructions are correct and we know them. Maybe I wasn't clear enough with my question.

Our client run Datastage Jobs by using Run Bash Script activities in Watson Pipelines Jobs.

To do this, include CPDCTL DSJOB RUN commands within the Run Bash Script activities. this works fine, no problem

The problem is to verify the Datastage Job finish status given that the CEL functions in expressions builders used in follows links apply to the activity status, not the Datastage Job finish status. (or at least that's what we were able to verify, where Datastage Jobs fails but the run bash activity that contains it does not fail, then the results status return ok).

Thanks in advance.

Javier.

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------

Thanks Thomas,

you are correct

cpdctl dsjob run -w -1 -p PROJECT_NAME -n JOBNAME

Will wait (-1 wait for an infinite amount of time) for the job to finish and return the Status code as RC.

You can run with -wait that will gives you a return code.

• 0: successfully completed
• 1: completed with warnings
• 3: completed with error
• 4: failed
• 5: canceled
• -1: other

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.4.8.3.md#jobs

Hi Javier,

so why do you just don't use a "Run DataStage/Pipeline job" activity to run your jobs in the pipeline? Is it because you want dynamically address the Jobname? If you don't use the "Run DataStage/Pipeline job" activity, then you would have to wrap the CPDCTL DSJOB RUN command into a skript that handles everything, I did this years ago in legacy DataStage with dsjob run, first checking the state of an DataStage job, resetting it if required, then run it and in the end again checking the state of the last run to check if it has been successfull.

I had a quik look at this for CP4D/CPDCTL

Command reference

https://github.com/IBM/DataStage/blob/main/dsjob/dsjob.5.0.0.md

To Run a job
cpdctl dsjob run -p PROJECT_NAME -n JOBNAME
This should always give you RC=0 if you are able run the job (e.g. you are able the connect to the prject and it exists) and will print a Status code to STDOUT which you can then parse.

In addition, you can run e.g.

cpdctl dsjob list-job-status -p PROJECT_NAME -n JOBNAME

afterwards, which will give you the Run Status of your Job.

I hope this helps.

Dear Ralf, thank you for you response and explanations.

Your instructions are correct and we know them. Maybe I wasn't clear enough with my question.

Our client run Datastage Jobs by using Run Bash Script activities in Watson Pipelines Jobs.

To do this, include CPDCTL DSJOB RUN commands within the Run Bash Script activities. this works fine, no problem

The problem is to verify the Datastage Job finish status given that the CEL functions in expressions builders used in follows links apply to the activity status, not the Datastage Job finish status. (or at least that's what we were able to verify, where Datastage Jobs fails but the run bash activity that contains it does not fail, then the results status return ok).

Thanks in advance.

Javier.

Hi Javier,

what is exactly your question? You use CPDCTL to start the pipelines, CPDCTL will give you a returncode for this, also inside the Watson pipelines you can control the success of a Flow or Pipeline run in it with a condition on the link exiting this activity (double click on the link, this is different to legacy where you would do this by opening the activity).

BTW. CP4D 5.0.0 is already there, it has some great improvements.

https://medium.com/@rpham_78948/ibm-cloud-pak-for-data-v5-0-datastage-user-experience-new-features-and-improvements-022a89e9b3b9

------------------------------
Ralf Martin
Principal Consultant
infologistix GmbH
Bregenz

Original Message:
Sent: Thu August 08, 2024 02:02 PM
From: Javier Pugliese
Subject: Get finish status DS Job executed in Run Bash Script Watson Pipelines

Dear all,

We are migrating Job Sequences IIS 11.5 to CP4D 4.8.5 Watson Pipelines.

Our client uses Run Bash Script activities to run Datastage Job using CPDCTL DSJOB RUN command

Are there some CEL functions to evaluate the subsequent result of the execution of the Job ?

What is the appropriate way to analyze this in conditions after that activity?

Javier.

------------------------------
Javier Pugliese
jpuglies@ar.ibm.com
------------------------------