HSTS is a licensed product, yes. It has additional components that are installed along with ascp (either asperatrapd or libpvcl_cloud) that allow ascp to interface directly with S3.
Mode recv means ascp will do a download, so your source is the path in the server and your destination the path (or in your case the s3 bucket)
ascp --mode recv --user xfer --host hostname /remote/source/path /local/dest/path
Mode send means ascp is doing an upload to the remote server, so your source would be a local path and your destination a path in the destination server:
ascp --mode send --user xfer --host hostname /local/source/path /remote/dest/path
------------------------------
Jose Gomez
------------------------------
Original Message:
Sent: Fri June 07, 2024 12:00 PM
From: Santiago Revale
Subject: Copy data from remote Aspera server to AWS S3 bucket
Hi Jose,
Is HSTS freely available or do I have to get a license to install/use it?
The EC2 I'm working from has a trusted relationship with the destination bucket but none of the commands I tried are working, whether I use "mode recv" or "mode send" (which by the way, what's the difference?) nor providing or not the AWS credentials.
Additionally, if I use the "--host MY_ASPERA_HOSTNAME /path/to/folder", I get a "Session Stop (Error: No such file or directory)" error, while if I omit the "--host" flag and I join the two with a ":", it works (when copying it locally).
------------------------------
Santiago Revale
Original Message:
Sent: Fri June 07, 2024 11:25 AM
From: Jose Gomez
Subject: Copy data from remote Aspera server to AWS S3 bucket
Hi Santiago,
For ascp to be able to use S3 natively in the way you describe you would need a full High Speed Transfer Server. This would be deployed in a EC2 instance in the same region as your destination bucket for optimal performance. The HSTS install will need to be configured for cloud transfers: https://www.ibm.com/docs/en/ahts/4.4.x?topic=linux-configuring-cloud.
Then you could do a download to your EC2 machine where the destination is your bucket. If you define a trust relationship between your EC2 system and the destination S3 bucket then you wouldn't need explicit credentials in the s3 URL:
ascp -P 33001 -p -k1 -l400m --mode recv --user USERNAME --host ASPERA_HOSTNAME /path/to/folder s3://ACCESS_ID:SECRET_KEY@s3.amazonaws.com/my_bucket/
Regards,
Jose
------------------------------
Jose Gomez
Original Message:
Sent: Fri June 07, 2024 06:51 AM
From: Santiago Revale
Subject: Copy data from remote Aspera server to AWS S3 bucket
Dear Aspera community,
I was given access to an Aspera server from where I need to download 63 TB of data. However, I would like to copy them directly to an AWS S3 bucket. I tried using the command "ascp" from Aspera Connect according to the documentation, but I'm getting an error. Below are the commands I tried and errors I've got:
# From an EC2 instance without specifying credentialsascp -P 33001 -p -k1 -l400m --mode send \ --user USERNAME \ ASPERA_HOSTNAME:/path/to/folder \ s3://s3.amazonaws.com/my_bucket/
# error message
Session Stop (Error: Session initiation failed, fail to set pvcl)
# From an EC2 instance providing credentialsascp -P 33001 -p -k1 -l400m --mode send \ --user USERNAME \ ASPERA_HOSTNAME:/path/to/folder \ s3://ACCESS_ID:SECRET_KEY@s3.amazonaws.com/my_bucket/
# error message
ascp: cannot copy from one remote host to another
Startup failed, exit
Thank you very much in advance for your help.
------------------------------
Santiago
------------------------------