Managed File Transfer

 View Only

  • 1.  Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Fri August 11, 2023 12:36 PM
    Edited by Andres Parada Mon August 28, 2023 03:45 PM

    In our environment, currently all the infra is hosted on On-Prem VMs, we are planning to migrate complete infra which is hosted on On-prem to AWS Cloud.

    We choose to go with Connect Direct as the preferred file transfer mechanism to do this migration.

    I have query here:

    How to make sure that the file is visible to the target only after the file is fully transferred from the Source?

    What is the mechanism of enabling the client, who is listening to the target system, to find the file only once the transmission is fully completed in Connect Direct?



    ------------------------------
    John Kundimi
    ------------------------------



  • 2.  RE: Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Mon August 28, 2023 06:24 AM

    Each file transfer in Connect:Direct is done through a Connect:Direct Process script. You specify the source and destination file names in a COPY step within this process. 

    It is also possible to let Connect:Direct perform tasks before or after the transmission using RUN TASK/JOB steps, for example to rename or move the file.



    ------------------------------
    Volker Friedrich
    ------------------------------

    Original Message


  • 3.  RE: Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Mon September 18, 2023 08:18 AM

    Hello Volker Friedrich,

    Thanks for your reply,

    I have a query here regarding the response received for File transfer in Connect Direct

    "

    Each file transfer in Connect:Direct is done through a Connect:Direct Process script. You specify the source and destination file names in a COPY step within this process. 

    It is also possible to let Connect: Direct perform tasks before or after the transmission using RUN TASK/JOB steps, for example to rename or move the file."

    How having this file name will help us to ensure or guarantee that the file transfer is complete ?

    Can you please share more details around it how do we do this file transfer?

    Thanks
    John Kundimi



    ------------------------------
    John Keerthi Kundimi
    ------------------------------

    Original Message


  • 4.  RE: Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Tue September 19, 2023 04:54 AM

    Where do you need the data to be? S3, EFS, EBS?

    For EFS or EBS you'd need an EC2 VM with a C:D node installed and the EFS or EBS volumes mounted and it's not really any different to an on-prem server/VM. At the end of the copy you can do a RUN TASK to rename/move the file, send a notification etc.

    If you need the data in S3 then you have 2 choices:
    1) have your on-prem C:D Linux node write directly to the S3 bucket as the destination
    2) install a C:D node on an EC2 Linux VM and specify the S3 bucket as the destination - better option as the data moves over the internet using C:D protocol instead of HTTPS with checkpoimting, compression etc.
    S3 will make sure that the file is not visible until finalised - it's part of the REST APIs used to write the file to S3 - and you can either setup notifications on the bucket for new files using Lambda, SNS/SQS etc. or use a RUN TASK/JOB to send a notification via REST (e.g. curl command) or however you need.

    Take a look at https://www.ibm.com/docs/en/connect-direct/6.3.0?topic=v63-using-object-store-providers-connectdirect-unix and https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

    Best regards,



    ------------------------------
    RICHARD CROSS
    ------------------------------

    Original Message


  • 5.  RE: Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Fri September 29, 2023 02:30 PM

    Hello John, 

    Let me see if I understand your question...

    Each transfer made through Connect Direct has end-to-end traceability.

    In other words, every transfer carried out via Connect Direct generates a process number, this process number when consulted will give you full visibility of the file copying process.

    There is a step called CTRC which is precisely copy control, through the result of this step it is possible to guarantee and visualize the exact moment that the file was copied to the other side and whether it was successful or not.

    It is also important to emphasize that Connect Direct for Linux has direct integration with s3 buckets, so it can read and write directly to an s3 bucket.

    Hope this helps.



    ------------------------------
    Felipe Ferraz
    ------------------------------

    Original Message


  • 6.  RE: Do a File transfer from On-Prem to AWs Cloud using Connect Direct

    Posted Mon October 02, 2023 03:57 PM

    Hi John,

     

    Complementing Felipe's explanation, below is a example of the succesful copy result.

     

    ===============================================================================

    ----------------------------------------------------------------------------------------------------------------------

    PROCESS RECORD   Record Id =>  CTRC

    Process Name       => sample         Stat Log Date  => 10/02/2023

    Process Number     => 483            Stat Log Time  => 11:44:38.902

    Submitter Class    =>

    Submitter Id       => cdadmin@cdunix61

     

    Step Start Date  => 10/02/2023     Step Start Time  => 11:44:38.795

    Step Stop Date   => 10/02/2023     Step Stop Time   => 11:44:38.902

    Step Elapsed Time=> 00:00:00

     

    Step Name        => step01

    From node        => P

    Rstr             => N

    SNODE            => cdunix61

    Completion Code  => 0

    Message Id       => SCPA000I

    Short Text       => Copy step successful.

    Ckpt=>Y  Lkfl=>N  Rstr=>N  Xlat=>N  Scmp=>N  Ecmp=>Y  Ecpr=>79.19 CRC=>N

    FASP=>N

    Zlvl=>1  Zwin=>13  Zmem=>4

    Local node       => P

    From node        => P

    Src  File        => /ibmcd/cdunix/ndm/cfg/msgfile.cfg

    Dest File        => /ibmcd/cdunix/work/cddelete.me

         Source                     Destination

    Ccode      =>0              Ccode        =>0

    Msgid      =>SCPA000I       Msgid        =>SCPA000I

    Bytes Read =>4725021        Bytes Written=>4725021

    Recs  Read =>206            Recs  Written=>206

    Bytes Sent =>983414         Bytes Recvd  =>983414

    Rus   Sent =>19             Rus   Recvd  =>19

    Ru    Size =>65536

    -------------------------------------------------------------------------------

    ===============================================================================

     

    Remembering that everything that is executed successfully has a return code equal to zero.

     

    Regards,

     

    Ricardo Pereira

     




    Original Message