Aspera

Aspera

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Aspera transfers bring our entire network down to its knees

  • 1.  Aspera transfers bring our entire network down to its knees

    Posted Fri December 12, 2025 11:38 AM

    We have 5Gbps symmetrical DIA internet, and a Watchguard enterprise router/firewall.  When a user runs Aspera transfers (sometimes but not always), our entire internet connectivity greatly suffers. A regular speed test doesn't reveal much, but:

    • Other users complain about sporadic slowness
    • We use plenty of remote access (WebRTC-based) which is sensitive to latency and jitter, and the quality of those connections tanks
    • packetlosstest.com on other systems returns awful results, with many late and/or dropped packets. When we stop Aspera, the results are fine again.
    • Today we had an internet outage for a few minutes, suspiciously almost perfectly time-correlated… minutes after I installed Aspera for a new user, who then started several downloads.

    Also:

    - This affects our entire LAN, regardless of which LAN segment (switch path) the Aspera user is on

    - The clients are on 1Gb LAN links, so no matter what, it's not nearly enough to saturate our internet bandwidth. Our router's CPU never goes over 30% either.

    - I haven't observed such behavior or network issues otherwise; we use Signiant services regularly too, and they don't cause such issues.

    - Aspera's transfers graphs are very jittery when this happens, sometimes with reconnects. Examples:


     

    When it's behaving, the line is smooth as butter:

    Without further help, what's left in my arsenal is to apply traffic shaping, congestion control, AQM, QoS, etc. to Aspera traffic… but isn't the basic guideline to not do that? "Aspera doesn't like traffic shaping, let it police itself" is what I'm used to (or from general UDP accelerators knowledge), but the current situation is untenable. 

    We don't have an Aspera server, so I don't have access to the official Aspera support channels - our clients` clients would be the ones on the server end of those transfers, and so I don't realistically see establishing a proper line of support to dig into this. Hence posting here.



    ------------------------------
    Drew Lahat
    ------------------------------


  • 2.  RE: Aspera transfers bring our entire network down to its knees

    Posted Fri December 12, 2025 01:23 PM

    Aspera transfers, even on client side, can be regulated with so called "V-links" or "trunks", but that assumes users would follow the rules...

    IMHO, since you don't have a server, traffic shaping with a Delay policy on UDP/33001 should allow you to limit Aspera traffic.

    The reason is that Aspera transfers (UDP) are regulated based on evolution of latency.

    At the beginning of a transfer a base latency is measured, and when congestion happens somewhere, latency increases, and IBM Aspera FASP takes this as indication to slow down.

    Packet drop, though, is not a good policy for Aspera...



    ------------------------------
    Laurent Martin
    ------------------------------



  • 3.  RE: Aspera transfers bring our entire network down to its knees

    Posted Fri December 12, 2025 03:19 PM

    Thank you, Laurent.

    What explanation would there be for congestion to exist, though? Considering that

    • The client's port speed is 1Gbps, the core switch & uplinks are 10Gb, and the internet bandwidth is 5Gb. 
    • I haven't applied any traffic shaping (yet)

    On the user's system, Aspera and WebRTC do compete with each other. The user has e.g. ~900Mbps of internet bandwidth, and the WebRTC remote access tool takes 1-15Mbps (varies continuously based on instantaneous use). Aspera of course is trying to use up all it can, with the default settings. But that WebRTC is <2% of the available bandwidth; is that enough to disturb Aspera so drastically to see the graph spikes I attached? Or conversely, what is a typical explanation for those graphs?
    And it still doesn't explain how and why this affects traffic upstream from that system whereas there's plenty of available bandwidth..



    ------------------------------
    Drew Lahat
    ------------------------------



  • 4.  RE: Aspera transfers bring our entire network down to its knees

    Posted Mon December 15, 2025 04:57 AM
    Edited by Michael Hoffmann Mon December 15, 2025 05:30 AM

    That's right, Laurent's comment doesn't explain why you're having this problem. But I'm also at a loss. I've never seen problems like this before. (taking all your information into account) Normally, Aspera is a relatively friendly protocol on the network with our adaptive traffic control algorithm.  

    ( Typical customer complaints, similar to yours, are of the sort  "we started a download with 5Gbps (ascp -l 5G ...), but it reached only 900Mbps. So we changed the ascp policy to fixed! now we have really high packet loss and our intenet link, is really slow. Our internet link is 1Gbps, and we thought Aspera is 5x faster" ;-) ) 

    In contrast, your description of the problem is very good. You are right that such problems are not part of our claim of how Aspera should work. But its hard to give a good guess without more info.

    My first guess was  "somewhere a DoS protection kicks in"  (maybe at the internet provider)  for the Aspera 33001 udp port.  

    Are all the (problematic) Aspera downlaods from the same server site?   

    Can we get receiver logs?  (Transfer logs from the client on download)

    Can you also try to download some test files from one of our demo server ?  

    I'm sure we can fix this.  



    ------------------------------
    Michael Hoffmann
    ------------------------------



  • 5.  RE: Aspera transfers bring our entire network down to its knees

    Posted Mon December 15, 2025 05:12 PM

    Hi Drew,

    To add more detail, the IBM Aspera FASP protocol will transfer at the specified target rate. The target rates can be set in the IBM Aspera application and in the latest releases will limit target rate for all transfers using the vlink feature mentioned by Laurent. This will happen per client machine although many clients are on the same network segment, it can use ip multicast to have multiple client using the same vlink parameters. From the screenshots, it looks like your clients are using IBM Aspera Connect which only has a per transfer limitation. One way to work around that is to use the Queue feature (in the Transfers tab) and limit the Queue to 1 transfer and limit the rate (in the Bandwidth tab).

    The network rate used by Aspera is based on fluctuations in round-trip-time (RTT) measurements (by default, there are different rate modules that have different properties) and the fluctuations in RTT will correspond to the network rate changing and never going higher than the target rate. Increases in RTT are interpreted as congestion in the network which cause IBM Aspera FASP to slowdown and decreases in RTT are interpreted as available bandwidth and cause IBM Aspera FASP to speed up.

    The graphs that you have posted are going to correspond to significant variation in RTT. Looking at the aspera-scp-transfer.log files would help to diagnose the problem. You can pull out all of the lines that have "Sender bl", "Receiver bl" or "FASP Session Statistics" in them and paste them here, our team can analyze and get a better understanding of what is happening over time. These lines will not have any personal information in them.


    The logs will have numbers that correspond to the different rate calculations and we can use that to help determine which part of transfer might be constrained. Those numbers correspond to network, storage and vlinks.

    Typical causes of rate problems are packet inspection devices, firewalls, storage, client or server cpu and/or network policies.



    ------------------------------
    BEN FORSYTH
    ------------------------------



  • 6.  RE: Aspera transfers bring our entire network down to its knees

    Posted Mon December 15, 2025 10:31 PM

    Thank you Ben, Pam, Michael. I appreciate the detailed responses. Since the cause/fix may be the router, core switch, and/or the ISP, getting more clarity from the logs on what's happening would definitely help.

    The user's aspera-scp-transfer.log from that day spans 4 hours and is 1.7MB. bl and stats alone add up to 1,560 lines.

    Is there a way to send private messages, to share the log file privately? I just need to share this without exposing client info in the open.



    ------------------------------
    Drew Lahat
    ------------------------------



  • 7.  RE: Aspera transfers bring our entire network down to its knees

    Posted Mon December 15, 2025 06:08 PM

    Hi All, Just noticed this interaction.  The only time I have ever noticed a behavior where Aspera transfers were interfering with other TCP based traffic was when routers had extremely low queuing/cache size.  This would cause a lot of discards in the Aspera transfer session which Aspera looks at as just life when working in a WAN.  So, Aspera would continue to transfer at as close to target rate as it could, but in reality there was congestion that couldn't be sufficiently handled by the routers in order for the roundtrip time on the transfer to go up and drive Aspera to reduce its speed.  These discards would affect TCP traffic as well which would make their performance horrendous.  



    ------------------------------
    PAM CURRIE
    ------------------------------



  • 8.  RE: Aspera transfers bring our entire network down to its knees

    Posted Tue December 16, 2025 04:23 PM

    Michael, I can send you the scp-transfer log privately. Just sent it to Ben Forsyth.

    Thank you.



    ------------------------------
    Drew Lahat
    ------------------------------



  • 9.  RE: Aspera transfers bring our entire network down to its knees

    Posted Thu December 18, 2025 01:00 AM

    Hi Drew,


    We were able to analyze the logs that you sent us and it looks like the maximum transfer rates that your client can sustain is about 200 Mbps.

    The constraints look to be primarily on the network. The network constraint looks to be either a firewall or rate policy that starts dropping all packets once there is more than 200 to 300 Mbps of traffic. There are also warnings in your logs about disk write speed being exceeded. This constraint happens when network is faster than the disk. There is a way to test for disk constraint, more details below.


    The target rate being used by the Aspera software is 10 Gbps and setting the rate this high is not allowing the software to change rates in a meaningful way when changes in round trip time are detected. The 10 Gbps rate was set by the end user using the Aspera Connect application. The server being used did not have a transfer limit set which allowed this to happen. I would recommend setting the maximum transfer rate to 200 Mbps until the root cause of network drops can be determined.
    In Aspera Connect, click on the IBM Aspera Connect menu and select Preferences.. then click the Bandwidth tab. Check the box below Downloads and set the limit to 200 Mbps. Then have your client run their transfers again.


    To help determine why 1 Gbps is not attainable you can use one of the test Aspera servers on the internet. You can run test commands with the fixed transfer policy to check what transfer rate your network can sustain by checking for packet loss. Using the fixed policy disables Aspera's rate controller and is only meant to be used for troubleshooting network performance.
    The ascp command line tool will be located inside of the IBM Aspera Connect.app bundle in the Contents/Resources directory

    Downloads - starts 300Mbps and increase and decrease by 1/2 until (300 -> 150 -> 225 -> 250, etc..) you see no packet loss.

    
ascp -l 300M --policy fixed -L- --mode recv --host demo.asperasoft.com --user aspera --no-write aspera-test-dir-large/1GB /tmp/

    This command will not write to disk because of the --no-write option. The logs will print to the Terminal screen. Cancel the transfer after 10 seconds to 20 seconds or let it complete and check the line with FASP Session Statistics and look at the effective= number and you want to see that number as close to 100% as possible. The example below is using 500 Mbps with a 100% effective rate.

    There is another command line option to check that might influence the rates. It is the -Z argument which can change the packet size. On macOS systems, only the root user can turn of the do not fragment bit which is used to figure the packet size the network can sustain. You can add -Z 1200 and see if there is a significant change in loss.

    LOG FASP Session Statistics [Sender] id=2e6909d1-fcf8-4583-af68-b82286fe2125 delay=6ms rex_delay=11ms solicited_rex=0.00% unsolicited_rex=0.00% ave_xmit_rate 485.94Mbps effective=100.00% effective_rate=485.94Mbps (detail: good_blks 744728 bl_total 744730 bl_orig 744729 bl_rex 1 dup_last_blks 1) (sndr ctl: sent 19 rcvd 19 lost 0 lost 0.00%) (rcvr ctl: sent 331 rcvd 331 lost 0 lost 0.00%) (rex  ctl: sent 1 rcvd 1 lost 0 lost 0.00%) (progress: tx_bytes 1048576012 file_bytes 1048576012 tx_time 17944290) (estimate transfer size: 1048576012 bytes)  sess->instru.num_rex_xfer_not_found 0 sess->instru.num_rex_abrtxfer_not_found 0


    Once you figure out the ideal rate that your network can sustain, you can look to remove the no-write option and confirm that your client machine disk's can also sustain the rates.

    Ideally, the Aspera software should automatically converge to the rate that the network is allowing but they do not work well when the target rate is very high compared to the actual rate that is being allowed. Next problem is that right away, there is excessive packet loss and the loss is constant where all attempts to retransmit packets will lose packets.

    Please let us know if either suggestion of limiting the rate or changing the packet size helps with efficiency. We can also schedule a call together to work through the testing to attempt to figure out what is happening.

    Thanks,

    Ben



    ------------------------------
    BEN FORSYTH
    ------------------------------



  • 10.  RE: Aspera transfers bring our entire network down to its knees

    Posted Fri December 19, 2025 11:41 PM

    Many thanks, Ben. I'm not aware of configuration on our router/firewall that would cause this, but of course I was, this wouldn't be happening in the first place. :-) I agree it's likely that the cause is in the router. The detailed information and instructions you provided give me excellent avenues for the testing and diagnostics work.

    Regarding the cap and 200Mbps speeds, the white graph I originally attached was from the same incident as the logs (timestamp was 2025-12-11 17:54:36). It shows the user-set cap was 915 and 904 Mbps, not 10Gbps.  And the effective speed was roughly 360Mbps, for each of 2 simultaneous transfer. Does that agree with the logs?

    I will follow the instructions and run controlled tests next week, and will give further updates.

    Thank you again and Happy Holidays!



    ------------------------------
    Drew Lahat
    ------------------------------



  • 11.  RE: Aspera transfers bring our entire network down to its knees

    Posted Mon December 22, 2025 08:49 PM

    Hi Drew,

    It would not be the first time I mis-read the statistics and mis-counted the zeros, so I pulled the rate from the FASP Transfer Stop lines to make sure it was 10 Gbps and they were all 10 Gbps

    2025-12-11 17:46:00.883 [1c8-17572590] LOG FASP Transfer Start uuid=728a5ad6-5d92-4010-80fa-8e58d2c7e095 op=recv status=started file="file name removed" size=size-removed-to start_byte=0 rate=10.00Gbps loss=0.00 rexreqs=0 overhead=0 mtime="2022-06-14 12:01:12"

    We have some scripts that attempt to aggregate the logs rate lines and saw the actual bytes transferred by reading the total bytes aggregated over an approximate interval of 60 seconds and see that the final transfers were in the 600 to 700 Mbps range.

    thanks and have a very happy holiday!

    Ben



    ------------------------------
    BEN FORSYTH
    ------------------------------