App Connect

 View Only
Expand all | Collapse all

SFTP node in ACE SaaS

  • 1.  SFTP node in ACE SaaS

    Posted 19 days ago

    I need to push and pull files from an SFTP server from ACE SaaS. I have few queries on this. Grateful if anyone can clarify

    Is there a SFTP node to push and pull files from an SFTP server in App Connect Enterprise SaaS version? 

    or do I you a file node in ACE Toolkit which that allows SFTP and deploy the message flow on ACE SaaS to achieve the same result?



    ------------------------------
    SudhakarM
    ------------------------------


  • 2.  RE: SFTP node in ACE SaaS

    Posted 18 days ago

    I have a pertinent query, as I'm grappling with a similar issue. Currently, I'm employing a File Output node to address this challenge. Essentially, it enables me to retrieve a file from a specified path on my local disk. Subsequently, leveraging the same configuration, I transmit it via FTP to a remote server.

    Presently, my investigation pivots towards exploring the inverse scenario. Specifically, I aim to ascertain if it's feasible to read from an SFTP route and retrieve the file(s) it encounters, subsequently saving them onto my local drive.



    ------------------------------
    [Armando] [Ruiz]
    [Madrid] [Madrid]
    armando
    ------------------------------

    Original Message


  • 3.  RE: SFTP node in ACE SaaS

    Posted 18 days ago

    Hi Sudhakar & Armando,

    ACE SaaS providing the file operations to move it from one location to another locations(FTP/SFTP). To do this we have to use the file input/ output/ Read nodes.

    and configure the FTP/SFTP server information in the file node properties and apply the setdbparms in the command for the run time.

    Note1: in SaaS environment use file mount configuration to apply setdbparms command.

    Note2: Use the base input directory as /tmp in the Basic properties of the file nodes. (/tmp has all the default previlizations).



    ------------------------------
    [Rakesh][Sudhagoni]
    [Middleware Developer]
    ------------------------------

    Original Message


  • 4.  RE: SFTP node in ACE SaaS

    Posted 17 days ago

    Please see my other comment on this post regarding SFPT use-cases. But specifically with regards to saving files to local drive within SaaS (and the same applies for our container offerings) the persistence is ephemeral - so you can use the local file system for transient storage in support of your message flow, but if the runtime stops for any reason (updated, moved, crashed, etc.) then the data would not persist.

    Please reach out if you have any questions.



    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message


  • 5.  RE: SFTP node in ACE SaaS

    Posted 17 days ago

    Hello

    For SFTP use-case you would use the File nodes within Toolkit and deploy the flows to App Connect Enterprise as a Service:

    • https://www.ibm.com/docs/en/app-connect/12.0?topic=nodes-fileinput-node
    • https://www.ibm.com/docs/en/app-connect/12.0?topic=nodes-fileoutput-node

    If you need to create any directory structures within /tmp then you can use startup scripts provided as generic file configurations (https://www.ibm.com/docs/en/app-connect/saas?topic=types-generic-files-type) and reference these in the StartupScripts section of the server.conf.yaml (https://www.ibm.com/docs/en/app-connect/saas?topic=types-serverconfyaml-type) if required.

    If you have any questions please reach out.



    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message


  • 6.  RE: SFTP node in ACE SaaS

    Posted 17 days ago

    Hello, these are the nodes utilized in our setup. However, during the configuration of these nodes, I've observed a particular behavior: when a file is deposited into the local disk path, the system promptly retrieves it and transmits it via sFTP. Conversely, our desired operation is the reverse: to read from an sFTP location periodically, and upon discovering a file within that directory, to transfer it onto our local disk.



    ------------------------------
    [Armando] [Ruiz]
    [Madrid] [Madrid]
    armando
    ------------------------------

    Original Message


  • 7.  RE: SFTP node in ACE SaaS

    Posted 14 days ago

    @armando ruiz - could you please describe what you are trying to achieve in a little more detail? Once you discover the file in the remote location what are you doing with it once you write it to local disk and why do you want to write it to local disk?



    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message


  • 8.  RE: SFTP node in ACE SaaS

    Posted 16 days ago
    Edited by Sudhakar smaddipatla 16 days ago

    Hi Martin, Rakesh

    Thanks for your response. I have found two issues, one each with fileInput and fileOutput nodes in SaaS environment.

     In this post, I will mention only fileInput node problem. For now, that is more important for me.

    FileInput node issue:

    The doco says "

    The path of the directory from which input files are processed. The directory must be in a file system to which the integration server has access. If the input directory does not exist, no files are processed. The FileInput node checks that the input directory exists at intervals that are defined by the Scan delay property. The input directory must exist, even if you are processing files over FTP, FTPS or SFTP.". 

    So, as per your suggestion, I used /tmp. However, it doesn't pic any files from the SFTP folder. 

    To test, I executed in toolkit with local filesystem folder C:\downloads as the folder in the Basic properties. It does create following folders:

    mqsiarchive
    mqsibackout
    mqsitransitin

    and tries to process files in the downloads folder rather than getting files from SFTP folder. 

    My queries are:

    1) n a SaaS environment, I don't have any visibility of folder structure of the OS. So, what folder can I supply in the Basic property?

    2) in local (toolkit) environment, why does it process folder in the "Basic" property rather than the SFTP folder?

    I checked the folder and files exist on the SFTP server manually logging in.

    Appreciate any help as it is first time I am working on ACE SaaS.

    regards

    SudhakarM



    ------------------------------
    Sudhakar smaddipatla
    ------------------------------

    Original Message


  • 9.  RE: SFTP node in ACE SaaS

    Posted 10 days ago

    I am now getting 'MAC or CIPHER not supported by the server.' error when reading the file (tried both FILEREAD and FILEINPUT nodes). I also checked that the SFTP server I am using does allow valid values (see below). I also checked that I can log into the SFTP server manually.

    There was a fix for v12 re this. https://www.ibm.com/support/pages/apar/IT39954

    Do you know if this fix applied to SaaS version?

    Are these values checked even if I am using userid/pw only instead of SSH?



    ------------------------------
    Sudhakar smaddipatla
    ------------------------------

    Original Message


  • 10.  RE: SFTP node in ACE SaaS

    Posted 6 days ago

    The ifix that you reference looks to have been targeted for inclusion in 12.0.3.0 of ACE. In ACEaaS we support the last two releases - so today you can select v12.0.10 or v12.0.11 when you create an integration runtime, or you can select v12.0 that would default to latest version (currently v12.0.11). There is a more recent APAR that does affect users of v12.0.11 and appears to reference the blowfish-cdc cipher specifically: https://www.ibm.com/support/pages/apar/IT45529. This is targeted for inclusion in v12.0.12 - but we should have picked up this iFix on v12.0.11 in ACEaaS.

    If I create an integration runtime using v12.0 you can see in the logs that (today) this results in a v12.0.11.3 runtime and you can see that the fix for IT45529 is applied:

    So if you are using the 12.0 or 12.0.11 version then you should have the iFix applied and you should be able to verify this for yourself in the logs.


    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message


  • 11.  RE: SFTP node in ACE SaaS

    Posted 6 days ago

    With regards to the folder structure under /tmp I would recommend that you create the folder structure to work within. To do this you will need to:

    1. Create a local file create-directores.sh with appropriate file mode (chmod 777):

    mkdir -p /tmp/IBM/FileInput/foo
mkdir -p /tmp/IBM/FileOutput/foo

    1. Create a ZIP file from the create-directories.sh named create-directories.zip

    2. Create a "Generic files" Configuration named create-directories and choose the create-directories.zip. Files are extracted to /home/aceuser/generic

    3. Create a "server.conf.yaml" Configuration named create-directories-config with the following content:

    StartupScripts:
  FirstScript:
    command: /home/aceuser/generic/create-directories.sh

    1. Create an IntegrationRuntime named create-directories and select the create-directories and create-directories-config Configurations

    You will now have the /tmp/IBM/FileInput/foo and /tmp/IBM/FileOutput/foo directory structures that you can work within and configure your File nodes within Toolkit to use.

    If you are unsure on how to work with files and SFTP then this section of the documentation is a good place to understand how this works: https://www.ibm.com/docs/en/app-connect/12.0?topic=applications-working-files, for example the sub-section for https://www.ibm.com/docs/en/app-connect/12.0?topic=files-reading-file-remote-ftp-ftps-sftp-directory.



    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message


  • 12.  RE: SFTP node in ACE SaaS

    Posted 5 days ago
    Edited by Sudhakar smaddipatla 5 days ago

    Thanks so much Martin for clear instructions. Both fileInput and fileOutput nodes work :-) using userid/PW authentication. 

    I encountered some issues while using create-directories.sh. While the script is placed as required in the folder using "Generic files" type, the startup keep failing with 32512 return code. Some posts told me it could be because of permissions. So, I started the server with chmod 777 on the create-directories.sh in the "server.conf.yaml". It allowed chmod on the file but it still failed with the same return code. So, I created folders one after the other instead of script file.

    I am now onto using SSH keys instead of basic auth.

    I can see doco says use e.g. "mqsisetdbparms -w c:\workdir\ACEServ1 -n sftp::identityD -u user4 -i C:\key_rsa_pp -r MyPassPhrase"

    How do I provide this in SaaS environment in the configs.

    Thanks again.



    ------------------------------
    Sudhakar smaddipatla
    ------------------------------

    Original Message


  • 13.  RE: SFTP node in ACE SaaS

    Posted 2 days ago

    I need some help in configuring SSH for SFTP server connection in ACE SaaS.

    What is the working directory of an integration server in ACE SaaS environment?

    How and where do I put the SSH private key file for SFTP connection?

    I generated the public key and shared with SFTP server and verified that I can login manually using the keys and passphrase.

    I tried various ways of putting the SSH key private key file in the working directory in ACE SaaS. From toolkit it is connecting ok. But I don't know the working directory of the integration server in a SaaS environment to put the SSH key file. 



    ------------------------------
    Sudhakar smaddipatla
    ------------------------------

    Original Message


  • 14.  RE: SFTP node in ACE SaaS

    IBM Champion
    Posted 2 days ago

    Have you tried deploying it via the bar file?



    ------------------------------
    Francois Brandelik
    ------------------------------

    Original Message


  • 15.  RE: SFTP node in ACE SaaS

    Posted 2 days ago

    Please see documentation around the available Configurations: https://www.ibm.com/docs/en/app-connect/saas?topic=information-configuration-types

    You can use specific Configuration types for providing:

    You can also use the Policy project Configuration type (https://www.ibm.com/docs/en/app-connect/saas?topic=types-policy-project-type) to provide an FTP Policy project (https://www.ibm.com/docs/en/app-connect/11.0.0?topic=properties-ftp-server-policy-ftpserver) separate from the BAR file if required.

    Each of these also provides details of where the files are extracted to within the local filesystem in case you need to reference them anywhere.

    For more "generic" files, or where there is no specific Configuration type we have the Generic files configuration type: https://www.ibm.com/docs/en/app-connect/saas?topic=types-generic-files-type

    "The Generic files type requires a .zip file of arbitrary files that are needed for custom configurations beyond the scope of types already provided. The .zip file contents are extracted into the directory /home/aceuser/generic, and can be referenced from other configurations or logic in the integration flows.

    The maximum recommended size of a .zip file for a configuration is approximately 660 KB."

    These should provide you the means to provide the resources you need and the documentation provides the locations you will find them on the local filesystem for referencing within you integration.



    ------------------------------
    Martin Ross
    IBM
    ------------------------------

    Original Message