Communications Server for Data Center Deployment

This problem occurs on zLinux with multiple CPUs due to frames getting placed out of order to the LLC2 driver. 

The Quick Beginnings Guide, Chapter 8, Configuring Ethernet LAN for Dependent and Independent LU, around page 67, has the answer. In RHEL 7, the kernel driver placed a frame order count on received packets so that the Ethernet LLC2 driver could put packets in order. When a system has multiple CPUs, most of the time, one CPU handles the Ethernet interrupts, but when things get busy, the other CPU will handle it and frames can get received out of order at the LLC2 driver level. 

The kernel removed this sequence count in RHEL 8 and later so that now the Ethernet configuration needs to be configured to mark one CPU as the interrupt handler. This will sequence the frames correctly.

Here is the configuration information from the documentation. This only applies to zLinux:

----  Start of Documentation ----
When using LLC2 connections on a System z Linux LPAR for systems other than RHEL7, if the LPAR has multiple CPUs, received packets can be placed into the SNA server's queue out of sequence from how
they arrived at the OSA adapter preventing the LLC2 link station from activating. In order to have the LLC2 packets received in sequence Receive Packet Steering (RPS) must be configured in Linux. Use the
rps_cpus bit table to indicate a single CPU to use for handling LLC2 packets per interface. To do this:

• Identify the interface name, this is usually something like eth1 or ensp0.
• Check the bit table, it should be all 0s, for example cat /sys/class/net/eth1/queues/rx-0/rps_cpus has an output of 00000000,00000000
• Set the bit in the table that maps to the CPU to use, in the example the first CPU, CPU_0 would be set to handle the LLC2 interrupts to queue packets using echo 1 > /sys/class/net/eth1/queues/rx-0/rps_cpus
• Make this RPS setting permanent by defining them in the Class Network Rules directory file /etc/udev/rules.d/70-persistent-net.rules by adding the string ATTR{queues/rx-0/rps_cpus}='1' to the interface network setting as in the following example of a qeth device number:

2f20 SUBSYSTEM=="net", ACTION=="add", Drivers=='qeth', KERNELS="0.0.2f20", ATTR{type}=="1",ATTR{queues/rx-0/rps_cpus}="1", KERNEL=="eth*", NAME="eth1"

-----  End of documentation -------

By masking the /sys/class/net/eth1/queues/rx-0/rps_cpus setting with bits, the Ethernet can handle sequencing the frames. 

Jeff L Smith

Hello, Jeff:

Thanks for your update.   My CSDCD configuration is running on x86 platform(RedHat over VMware), and the out of sequence frame is observed on the other side, a z/VSE platform using OSE and LLC2. 

Will it also possible if the RedHat is equipped with more than 1 CPU and the send out sequence is out of order?    

If the system is running in VMWare on x86_64, the RedHat will only send the LLC2 out the configured adapter, not a second adapter that may be configured for IP. However, VMWare has the bridge code built into its network configuration, and it is possible that the VMWare is sending the frames out of sequence. We recommend for VMWare systems, the LLC2 adapter be a dedicated LAN interface for the Guest that is not shared with any other guest on the system. Meaning it has direct access to the physical adapter. If the VMWare is configured to bridge the virtual adapter to a physical, then it may not be always sending on the one MAC source. Bridging LLC2 across Ethernet segments requires a logically configured set of bridge numbers that do not have loops possible. Direct access is the safer way to assure the LLC2 frames are going where they should be. 

Jeff L Smith