Aspera

 View Only
Expand all | Collapse all

Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

  • 1.  Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 19 days ago

    Hello there. I have looked at the existing blog post about API Connect and I don't believe it answers the question, however I stand ready to be corrected.

    I am developing an integration for a client that will require a file located in temporary Lambda storage to be delivered to a third party Aspera destination. I am not in control of that destination. I will only get credentials suitable for manual use in an Aspera Shares session, or in Aspera Client. I would like to have the Lambda function perform the upload.

    How would I go about making that happen? My current plan is to install the Aspera CLI as a Lambda layer - perhaps I have already hit on the best method although there do appear to be many APIs available that might work.

    All the best

    JS



    ------------------------------
    Jeremy Smith
    ------------------------------


  • 2.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 19 days ago

    Hi Jeremy,

    Using the Aspera CLI as you mentioned is the way to go.

    See this reference

    https://github.com/IBM/aspera-cli?tab=readme-ov-file#shares-1-sample-commands

    The ascli will make the API call into Shares to get the transfer token from the High Speed Transfer Server and then ascli will start the transfer.

    thanks,

    Ben



    ------------------------------
    BEN FORSYTH
    ------------------------------



  • 3.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 19 days ago

    Right, `ascli` can be a way to go.

    Note that there is also a container with `ascli`

    Another possibility is to develop a small code to use the Aspera Transfer SDK, one can find code here: https://github.com/laurent-martin/aspera-api-examples

    The Shares API for transfer is the same as the node api.

    Also, since ascli code is open source, one can be inspired by it .



    ------------------------------
    Laurent Martin
    ------------------------------



  • 4.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 17 days ago

    Thank you - would you mind giving me a hint as to which of the Python examples most closely matches my use-case? I will be uploading a few small xml and png files to Aspera Shares I believe. (I know, Aspera is massive overkill for that, but it looks as if it's a requirement of the third party recipient).

    My alternative plan (now I know about the container) is to set up the container image as an Amazon ECS/Fargate task and fire a command line string at it from a Lambda function. I can mount an EFS volume to both Lambda and ECS, so I can make the files accessible to both bits of code.

    Are you aware of this already having been done, and if so did it work?

    Thank you



    ------------------------------
    Jeremy Smith
    ------------------------------



  • 5.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 17 days ago
    Edited by Laurent Martin 17 days ago

    This sample should work with shares:

    https://github.com/laurent-martin/aspera-api-examples/blob/main/python/src/node.py

    In the configuration, the URL shall be like this:

    https://shares.example.com/node_api

    note the node_api at the end

    Then you also need to download the SDK:

    https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Introduction

    or , if you just clone the github repo, I believe it downloads it automatically from: https://ibm.biz/aspera_sdk

    If you want to try the CLI: https://github.com/IBM/aspera-cli, or even using the container of it, the command is:

    ascli shares --url=https://shares.example.com --username=xxx --password=yyy files upload --to-folder=/sharesname file1 file2

    (there are many options to create a config file, encrypt secrets, etc...)

    ------------------------------
    Laurent Martin
    ------------------------------



  • 6.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 15 days ago

    Thank you. I now have the container running in Amazon ECS, with a EFS volume mounted to it and also mounted to a Lambda function. I would like to test this by connecting to a Shares location and listing the files and folders already there (to confirm connectivity).

    Please could you tell me the ascli command I will need to run for that? I have looked and CTRL-Fd my way through the readme file and cannot anywhere find how to do this. ALternatively, if there is a less in-depth FAQ style documentation somewhere, please could you let me know where it is and I will be able to self-serve.

    All the best



    ------------------------------
    Jeremy Smith
    ------------------------------



  • 7.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 15 days ago

    There is a (short) section for Shares in the manual:

    https://github.com/IBM/aspera-cli?tab=readme-ov-file#shares-1-sample-commands

    As you can see, there is the browse command

    in the ascli shares command do browse /

    You can also populate a configuration file, if you need to execute several commands in a row and do not want to have a too long command line, you can do this:

    ascli conf preset update myshares --url=https://shares.example.com --username=xxx --password=yyy

    ascli conf preset set default shares myshares

    and then you can simply browse with:

    ascli shares br /

    (one can use just the significant prefix to shorten commands)

    The config file is kept in the user's home: $HOME/.aspera/ascli

    If you prefer, you can also use env vars:

    export ASCLI_URL=...

    export ASCLI_USERNAME=..

    etc...

    plenty of options to accomodate various use cases...



    ------------------------------
    Laurent Martin
    ------------------------------



  • 8.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 15 days ago

    Hello -

    I am very sorry bugging you about this. I wish I could work it out for myself. I have run the command you hinted at, and there is an error from ascli in the container in all these formats:

    ERROR: Argument: unknown value for command: shares --url=https://xxx.xxx.xx --username=xxx --password=xxx browse /

    ERROR: Argument: unknown value for command: shares --url=https://xxx.xxx.xx --username=xxx --password=xxx browse

    ERROR: Argument: unknown value for command: shares --url=https://xxx.xxx.xx --username=x -xx-password=xxx files browse

    Please can you advise? Once I get this working all I need it to do is send a few tiny files so nothing complicated. Thank you for pointing out the other documentation - I have now found the pdf file which has a helpful index.

    All the best,

    Jeremy Smith



    ------------------------------
    Jeremy Smith
    ------------------------------



  • 9.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 15 days ago

    This cmd works for me

    ascli shares --url=https://shares.example.com --username=XXXXXX --password=YYYYYY files browse /

    Can you post the exact cmd (without user/pass/host) you are using



    ------------------------------
    BEN FORSYTH
    ------------------------------



  • 10.  RE: Uploading to a third party Aspera Shares or Aspera Enterprise location from AWS Lambda

    Posted 15 days ago

    Hello -

    Thank you for replying. I have managed to get a command to run. The reason it was not working was that in the AWS Console ECS "run task" page there is a section for Container Overrides where the ascli command parameters can be entered. As I am new to Docker I did not realise that this needs to be a comma separated list of params. Once I formatted the command correctly, it worked and ascli continued to run with those params. It's now reporting connection refused on port 443 of the Shares endpoint, but I guess that's something I should take up with the third party.

    So now I think I have everything I need to continue.

    Thank you both very much for your assistance.



    ------------------------------
    Jeremy Smith
    ------------------------------