Aspera

Aspera

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
  • 1.  Migrate Massive data in one transfer session

    Posted 5 days ago

    we are planing to migrate a massive data from DC to DR using  2 HSTE servers

    uploading link between 2 sites is upto 100gbps

    I need to transfer 3PB per transfer session with 50gbps license tier

    please advice with hardware specs for the 2 HSTE to achieve this challenge 



    ------------------------------
    Ayman Fathy
    Cloud Solution Advisor
    ------------------------------


  • 2.  RE: Migrate Massive data in one transfer session

    Posted 4 days ago

    As of this writing the fastest speeds we see transferring with Aspera, using 1500 byte packet size is about 5Gbps. You will need a higher performance machine that can achieve clock speeds higher than 3.5 GHz with a minimum of 16 cores.

    To reliably transfer PB of data, you need to carefully orchestrate the file transfers and account for each file that has been sent and failed. Then to achieve higher than 5Gbps per ascp command, you will need to parallelize your transfers to achieve higher aggregate rates. The sizes of the files as well as the speed of the storage will greatly influence the rates that you can achieve. The the more parallel transfers, the higher number of cores you will need.

    This type of migration takes careful planning and testing and re-testing to validate your hardware and orchestration can achieve the amount of data you are looking to transfer. The process is typically, run a single transfer with the --no-read --no-write options to see the maximum single session rate, then remove the one of --no-read and confirm your rates with reading from storage. If reading happens with no slowdowns, then remove --no-write and confirm your rates don't change. Then add parallel sessions and confirm that your aggregate rates are the same as your single session rates. Iterate on these tests until you achieve the rates needed to get to 3 PB in the defined timeframe.

    As a reference, the math or time to transfer is 3 PB * 50 Gbps = 5.56 days. Realistic times would be to double that where your maximum transfer rate is 25 Gbps and time goes up to about 11 days.



    ------------------------------
    BEN FORSYTH
    ------------------------------



  • 3.  RE: Migrate Massive data in one transfer session

    Posted 4 days ago

    Thanks Ben 

    what about if I try to split the transfer session on 5 HSTE  with using Vlink

    is that better than using one HSTE in each site?

    Please Advice

    like the following diagram



    ------------------------------
    Ayman Fathy
    ------------------------------



  • 4.  RE: Migrate Massive data in one transfer session

    Posted 3 days ago

    Yes, adding more servers to the mix can help with parallelism.

    The exact number needed will depend on what speeds you can get out of a single session and then how many concurrent sessions can you run on a single machine. Then you can apply that knowledge to know the total machines you would need to complete the job.

    The Node API can accept payloads to run transfers. Each Transfer Server can connect to the same Redis database to allow all machines in the cluster to pull payloads out of the queue and run the transfers across all machines in the cluster.



    ------------------------------
    BEN FORSYTH
    ------------------------------