We have an ongoing FTP problem.  We are running two different RD&T systems on the same hardware.  The first is a two system sysplex utilizing 7 processors and 6 GB of memory.  eth1 (1 Gb) is shared with Linux.  The second is a single image with no sysplex utilizing 1 processor and 6 GB of memory and uses eth2 (also 1Gb).  eth1 and eth2 are connected to the same router. 

When we try to FTP PUT from a zEC12, zOS 1.13, 10 GB OSA system located in the same building, the FTP transfer rate is 3-7 Mb/second.  If the file is large enough, it fails.  A packet trace from the router connected to that MF shows a large number retransmits.

When we FTP PUT from a  zEC12, zOS 1.13, 10 GB OSA system located 1200 miles away (about 52 millisecond latency), we get 6000 to 9000 Mb/sec.

When we FTP PUT from our second RD&T system to our first one, we get 6000 to 9000 Mb/sec.

When we FTP from the zEC12 in the same building to another server (also 1Gb) attached to the same router as our RD&T systems, we get 60,000 to 80,000 Kb/sec.

In summary, the problem occurs for FTP PUTs only from the zEC12 in the same building.  It is like the RD&T systems are overrun and can't process the incoming packets quickly enough.  FTP GETs from the same system have the same 6000 to 9000 Mb/sec rate. 

Any suggestions how to alleviate this problem?

Hello. Would you please provide some background details on your machine and system setup? Specifically:

1. How are the two RD&T instances co-existing on the same hardware? Are you using a hypervisor for just one or for both RD&T systems? If so, which hypervisor (i.e. VMWare or KVM)?
2. What operating system is used?
3. How many physical cores (CPs) does the hardware machine have?

Thanks,

Stephen

Thanks for your reply.

1. The two RD&T systems are on tge same hardware.  No hypervisoris used.

2. From cat /proc/version:  Linux version 2.6.32-504.12.2.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-9) (GCC) ) #1 SMP Sun Feb 1 12:14:02 EST 2015

3. nproc command shows 64 processors.  (I think it is eight 8 core processorsa.)

Here is our hardware - HP DL580 with 19 cores and about 62 GB of memory

This is from page 148 of the zPDT Guide and Reference (SG24-8205-00):

7.5.9 Performance problems
At the time of writing, we were aware of two particular problems that affect OSA performance:
- If frames larger than expected are used, there might be an excessive number of frames
dropped (causing a retransmission). This might not be noticed unless careful
measurements or comparisons are made. We believe this problem is resolved by including
the systcl parameter that is now recommended:
 

           net.core.rmem_max=1048576

- If advanced Linux kernels are installed, there might be a drastic slowdown of OSA
performance that is immediately obvious. This is due to Linux attempting to offload
checksum functions into the adapter, which is not acceptable to the current awsOSA
implementation. One solution is to use a Linux command:

# ethtool -K eth0 rx off
# ethtool -K em1 rx off (newer style of NIC naming)

Unfortunately, this command must be entered after each Linux boot. The problem was first
noticed with a build of Linux kernel 2.6.36.2. The ethtool must be at least at level 2.6.33.
IBM has not published performance specifications for OSA. Informal observation indicates
that FTP throughput may be in the 5 - 8 megabytes/second range, assuming an
unconstrained network in a dedicated environment. If your performance is much worse than
this, the two problems mentioned here might be reviewed.

Thanks for your reply.

From sysctl -p command 
net.core.rmem_max = 1048576
net.core.rmem_default = 1048576

From ethtool -k command for both eth1 and eth2
rx-checksumming: off

I have tried FTPs with rx-checksumming on and off and I see no difference in the results.

Sherril, this comment is really only meant as an idea of what can cause an issue where an FTP works remotely and not from within the datacenter. I have seen this issue arise when TCP segmentation (or IP fragmentation) is taking place. In particular, if you are traversing 1200 miles, then there is the possibility a router in between is causing the packet size to be reduced, either through ip fragmentation or pathmtu. Within the datacenter, this may not be taking place, and you may be getting a much bigger frame size, quite possibly jumbo frames. The larger size might be leading to problems where the smaller size does not; for example, jumbo frames are not technically supported for zPDT (according to the redbook).  

If the zEC12, zOS 1.13 in the building with the RD&T instances is a test machine where you can try things, you could try setting the MTU on your route to the RD&T instances to 1492. Just an idea.

Thanks to Wilbert for notifying me that an April, 2015, version of the zPDT Guide and Reference (SG24-8205-01) was available.  In section 7.5.9 there are additional commands for NIC settings that "might improve the situation."

In addition to rx off, there are tso, gso, gro and lro off.  rxvlan is also listed, but we don' have any RD&T vlans.  I applied all of them and the transfer rate from the mainframe in the same building went from 3-7 Mb/sec to 39,000 Mb/sec.  I definitely consider a 10000x transfer rate an improvement!

I tested to see if there was one setting in particular.  While different combinations provided increased performance, the best performance was with tso, gso, gro and lro off.  Surprisingly, rx off, the only one documented in the old release seemed to have no effect on performance.  Also, I saw no significant change from the remote mainframe or from the other RD&T system with these parametes in place.

Thanks to everyone for your suggestions.