Data Integration

Data Integration

Connect with experts and peers to elevate technical expertise, solve problems and share insights.

 View Only

  • 1.  How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Mon June 10, 2024 10:50 AM

    HI, i was trying to run cat command to open a txt file(table_list_ALLGEMEIN_ALL.txt) in "Run Bash Script" Stage in a pipeline but i am getting error " No such file or directory". My file is present in a bucket in cloud object storage . I am giving following command

    cat Full_Bucket_name/DataStage/files/Pipeline/table_list_ALLGEMEIN_ALL.txt. 

    Does anyone face similar issue or sorted this out of how to give the correct Path of the file present in Bucket in COS?



    ------------------------------
    Sheikh Suleman
    ------------------------------


  • 2.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Tue June 11, 2024 03:43 AM
    Edited by Gerhard Paulus Tue June 11, 2024 03:44 AM

    Hi Sheikh Suleman,

    sharing files between DataStageFlows/-Jobs and Watson Pipelines is not that easy.

    You have to use PVCs and mounted in the DataStage and Pipeline Pods.

    https://www.ibm.com/docs/en/cloud-paks/cp-data/4.8.x?topic=pipeline-storage-data-access#sharing-files-between-datastage-and-pipelines

    I am not really sure if this works with IBM Cloud Object Storage (on IBM Cloud) and Run Bash Node within a Watson Pipeline ... I have to test it further.

    Regards,
    Gerhard



    ------------------------------
    Gerhard Paulus
    ------------------------------

    Original Message


  • 3.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Wed June 12, 2024 08:52 AM

    Hi Gerhard,

    Thank you for the info, I read the info on the link provided by you and tried some things but did not work though i found some more information here (https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-storage-data-access.html?context=wx), but it was also not helpful.



    ------------------------------
    Sheikh Suleman
    Frankfurt am Main
    ------------------------------

    Original Message


  • 4.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Wed June 12, 2024 01:38 AM

    Hi Sheikh,

    try the following, Do a touch in you "Run Bash Script" Stage and look where this lands. Afterwards you can try to write something via COS which you can access with both Flow and Pipeline.

    As Gerhard has already noted, Even in CP4D it is not that easy/intuitive to access the same files in Flows and Pipelines, but there we have Storage Volumes, which don't exist in DaaS.

    Like described in the Method above, I determined where my Files from the Sequential File stage where writen (I didn't like the COS stage back then), so I knew where to put them when I was going to read them.

    KR Ralf



    ------------------------------
    Ralf Martin
    Principal Consultant
    infologistix GmbH
    Bregenz
    ------------------------------

    Original Message


  • 5.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Wed June 12, 2024 08:55 AM

    Hi Ralf, thnk you for the tip. I was able to create a file using touch command but i tried to search for this file in my bucket in COS and i was not able to find where it landed.



    ------------------------------
    Sheikh Suleman
    Frankfurt am Main
    ------------------------------

    Original Message


  • 6.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Wed June 12, 2024 04:32 PM

    To access files in IBM Cloud Object Storage you need an IAM Token.  For cURL:
    curl "https://(endpoint)/(bucket-name)/(object-key)" -H "Authorization: bearer (token)"'

    https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-curl



    ------------------------------
    Thomas Krupski
    Analytics Service Cloud Pak for Data/WKC/IIS
    IBM
    ------------------------------

    Original Message


  • 7.  RE: How to get the correct path for files in bucket in IBM COS to use in "Run Bash Script" Stage in Watson Pipeline

    Posted Wed July 24, 2024 06:01 AM

    Hi everyone,

    Thank you for sharing your knowledge. I tried all things but cannot access the files directly from cloud object storage although i found kind of alternative solution. 

    In the above example i need to first download the file "table_list_ALLGEMEIN_ALL.txt" from IBM Cloud Object Storage using "cpdctl asset file download --path FILE_PATH" command into another file using "--output-file FILE_NAME" command and then i can use cat command to open the this file.So final command look something like this 

    cpdctl asset file download --path DataStage/files/Pipeline/table_list_ALLGEMEIN_ALL.txt --output-file table_list_ALLGEMEIN_ALL.txt
    cat table_list_ALLGEMEIN_ALL.txt

    Now this file can be accessed within this pipeline. I hope it helps.

    Regards

    Sheikh Suleman



    ------------------------------
    Sheikh Suleman
    Frankfurt am Main
    ------------------------------

    Original Message