AIX

ROCEv2 Enablement on AIX 

Contributors:  Vishwashreya Machineni, Swetha Venkannagari and Neelima Nandula

What is RoCEv2? 

A straightforward extension of the RoCE protocol enables traffic to operate in IP layer 3 environments.  This capability is obtained via a simple modification of the RoCE packet format. Instead of the GRH(Global Route Headers) used in RoCE, IP routable RoCE packets carry an IP header which allows traversal of IP L3 Routers and a UDP header (RoCEv2 only) that serves as a stateless encapsulation layer for the RDMA Transport Protocol Packets over IP. 

The ports of RoCEv2 adapters can be cabled via a router together by a switch. AIX OS supports both RoCEv1 & RoCEv2. RoCEv1 mode is default mode. 

List of adapters that support RoCEv2 

Prerequisites for RoCEv2 

The following are high-level function requirements for RoCEv2 support in OFED: 

• The user have the ability to select between two modes: RoCEv1(GRH legacy) and RoCEv2(UDP/IP encapsulation). 

• Only one mode can be active per AIX machine instance either RoCEv1 or RoCEv2 is active per instance. 

• The new OFED stack software is backward compatible with older Mellanox drivers. 

Support Matrix 

User RoCEv2 mode Selection: 

How to configure RoCEv2 on AIX? 

1.Set the RoCEv2 mode by using the command-  "no -r -o roce_v2_mode=1" and reboot the machine to take effect.

2.Check with “lsattr” what the current state of rdma for the port is. 

3.Enable the rdma attribute to “required” for the same port  

4. Configure the IP on the port and check ibv_devices list where it lists the RoCE enabled port. 

6.check the rping by running server & client as shown below.