Data Integration

 View Only

  • 1.  How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted 24 days ago

    HI, i was trying to run cat command to open a txt file(table_list_ALLGEMEIN_ALL.txt) in "Run Bash Script" Stage in a pipeline but i am getting error " No such file or directory". My file is present in a bucket in cloud object storage . I am giving following command

    cat Full_Bucket_name/DataStage/files/Pipeline/table_list_ALLGEMEIN_ALL.txt. 

    Does anyone face similar issue or sorted this out of how to give the correct Path of the file present in Bucket in COS?



    ------------------------------
    Sheikh Suleman
    ------------------------------


  • 2.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    IBM Champion
    Posted 24 days ago
    Edited by Gerhard Paulus 24 days ago

    Hi Sheikh Suleman,

    sharing files between DataStageFlows/-Jobs and Watson Pipelines is not that easy.

    You have to use PVCs and mounted in the DataStage and Pipeline Pods.

    https://www.ibm.com/docs/en/cloud-paks/cp-data/4.8.x?topic=pipeline-storage-data-access#sharing-files-between-datastage-and-pipelines

    I am not really sure if this works with IBM Cloud Object Storage (on IBM Cloud) and Run Bash Node within a Watson Pipeline ... I have to test it further.

    Regards,
    Gerhard



    ------------------------------
    Gerhard Paulus
    ------------------------------

    Original Message


  • 3.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted 23 days ago

    Hi Gerhard,

    Thank you for the info, I read the info on the link provided by you and tried some things but did not work though i found some more information here (https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-storage-data-access.html?context=wx), but it was also not helpful.



    ------------------------------
    Sheikh Suleman
    Frankfurt am Main
    ------------------------------

    Original Message


  • 4.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    IBM Champion
    Posted 23 days ago

    Hi Sheikh,

    try the following, Do a touch in you "Run Bash Script" Stage and look where this lands. Afterwards you can try to write something via COS which you can access with both Flow and Pipeline.

    As Gerhard has already noted, Even in CP4D it is not that easy/intuitive to access the same files in Flows and Pipelines, but there we have Storage Volumes, which don't exist in DaaS.

    Like described in the Method above, I determined where my Files from the Sequential File stage where writen (I didn't like the COS stage back then), so I knew where to put them when I was going to read them.

    KR Ralf



    ------------------------------
    Ralf Martin
    Principal Consultant
    infologistix GmbH
    Bregenz
    ------------------------------

    Original Message


  • 5.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted 23 days ago

    Hi Ralf, thnk you for the tip. I was able to create a file using touch command but i tried to search for this file in my bucket in COS and i was not able to find where it landed.



    ------------------------------
    Sheikh Suleman
    Frankfurt am Main
    ------------------------------

    Original Message


  • 6.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted 22 days ago

    To access files in IBM Cloud Object Storage you need an IAM Token.  For cURL:
    curl "https://(endpoint)/(bucket-name)/(object-key)" -H "Authorization: bearer (token)"'

    https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-curl



    ------------------------------
    Thomas Krupski
    Analytics Service Cloud Pak for Data/WKC/IIS
    IBM
    ------------------------------

    Original Message