Cloud Pak for Data

 View Only

 Accessing run logs in Watson Pipelines

Richard Lewis's profile image
Richard Lewis posted Tue October 21, 2025 08:47 AM

Hello, hoping to get some pointers to help me with this question.  My overall goal is to develop a custom error handler (notebook node) in Watson Pipelines, running on CP4D 5.1.3 (upgrading to 5.2.2 in a few months), that will parse the live consolidated log, searching for error messages from the most recent node, most likely be a DataStage node, and take some action.

I've started coding a python script (notebook node) as the error handler and I am able to authenticate and get a token, also on the verge of obtaining pipeline parameters and environment variables set in the pipeline. However, it is unclear how to access the live log.  I would appreciate any help that can be provided.

Thanks,

Rick Lewis

Azhagumuthu S's profile image
Azhagumuthu S posted Wed October 22, 2025 07:20 AM
Accessing run logs in Watson Pipelines can be accomplished through the IBM Cloud Pak for Data console or via the cpd-cli command-line interface.
Using the IBM Cloud Pak for Data Console:
  • Navigate to your pipeline: 
    Go to your project in Watson Studio or watsonx and locate the pipeline you want to examine.
  • View Run History: 
    Hover over the pipeline, select the three dots, and choose "View run history."
  • Go to Monitoring Hub (Optional): 
    For more detailed information and filtering options, select "Go to monitoring hub."
  • Select a specific run: 
    Choose the specific pipeline run you are interested in from the list.
  • Access Node Inspector: 
    If the pipeline is running or has completed, select a node on the canvas. Then, click the "Node Inspector" tab to view details, including logs, input, and output for that specific node's operation.
  • Consolidated Logs: 
    While the pipeline is running, you can scroll through consolidated logs to monitor progress.
Using the cpd-cli Command-Line Interface:
  • Authenticate: Ensure you are authenticated with the cpd-cli.
  • Get pipeline run logs: Use the cpd-cli pipeline run get-logs command. This command requires you to specify either the project_id or space_id where the pipeline resides.
    cpd-cli pipeline run get-logs --project_id <your_project_id> --run_id <your_pipeline_run_id>
You can also filter logs by stage ID or job ID within a specific pipeline run using additional flags, as described in the ibmcloud dev CLI documentation.

Related Content