IBM Z and LinuxONE - Group home

New with IBM z16: RoCE Express3 for optimizing performance

z/OS Communications Server|Erin ZHANG|May 18, 2022

  
IBM z16, the industry's first quantum-safe system, brings breakthrough technologies for AI and cyber resiliency to accelerate your digital transformation in hybrid cloud. In addition to its innovative on-chip AI and modernization advances, a new RoCE Express3 feature is also available on IBM z16. This technology refresh provides improved overall performance and Reliability, Availability, and Serviceability (RAS) capabilities.  
RoCE Express3 features, which support dual ports (10 GbE and 25 GbE), use Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) to provide fast memory-to-memory communications between two IBM Z platforms.

These features help reduce consumption of CPU resources for applications that use the TCP/IP stack. This provides significant performance benefits by offloading processing from the operating system.

They can also help reduce network latency with memory-to-memory transfers by using Shared Memory Communications over RDMA (SMC-R) for you to transfer huge amounts of data quickly and at low latency. SMC-R is transparent to the application and requires no code changes, which enables rapid time to value.

RoCE Express3 also supports RoCEv2 which enables SMC-Rv2 over multiple IP subnets. SMC-Rv2 extends the benefits of SMC-R and RDMA to additional z/OS workloads.  SMC-Rv2 requires z/OS V2R5.

If you’d like to configure and utilize the new RoCE Express3 features on z/OS, follow this link: https://www.ibm.com/support/pages/node/6564997. The features are now available on z/OS V2R3, V2R4, and V2R5 Communications Server with TCP/IP APAR PH34117 and SNA APAR OA60855. 
 
Best practice
z/OS Communication Server now supports the new RoCE Express3 features, in addition to the existing RoCE Express and RoCE Express2 features.
  • If you mix RoCE Express, RoCE Express2, and RoCE Express3 features in your SMC-R configurations, provision Peripheral Component Interconnect Express (PCIe) function ID (PFID) values with the same interface speeds on the TCP/IP definition.
  • When you configure your RoCE Express3 FIDs, remember to specify the port number in IOCDS/HCD.
  • If you configure VLANs in your SMC-R capable OSA interfaces, you must configure your RoCE switch ports to authorize all the VLANs associated with that physical network ID (PNetID).
  • If you specify an MTU size above 1000, you must configure jumbo frame support in your RoCE switch ports.
  • For high availability RoCE Express features should be provisioned to z/OS in pairs with each physical port having equal bandwidth.   
    To explore more about IBM Shared Memory Communications, check out this page: https://www.ibm.com/docs/en/zos/2.5.0?topic=communications-shared-memory-reference-information