Recommended GPFS network tuning parameters for large clusters on Linux

• GPFS: socketMaxListenConnections=maximum number of nodes in a cluster





• The OS limit must also be increased:
• OS: Linux: sysctl net.core.somaxconn=<maximum number of nodes in a cluster>


• OS: AIX: no -p -o somaxconn <maximum number of nodes in a cluster>







• GPFS: =0


• GPFS: =60


• GPFS: =60


• GPFS: =128


• GPFS: =number of logical CPUs on the node


• OS: Linux: sysctl net.core.netdev_max_backlog=250000


• OS: Linux: sysctl net.ipv*.neigh.*.=9 and/or net.ipv*.neigh.*.








• OS: Linux: sysctl vm.min_free_kbytes=an order of magnitude of 5-6% of the total amount of physical memory


• OS: Linux: modprobe.conf  ib_ipoib
  send_queue_size=8192 recv_queue_size=8192 (specific to Mellanox adapters - see Linux System Tuning for recommendations on ethernet RX / TX buffer tuning and Intel IB adapter tuning but note that page was based on work done in 2013)


• OS: Linux: IPoIB is often recommended to be run in datagram mode on larger systems, but please consult your network vendor for their recommendations on the IPoIB mode to use.
  
  Note that the Linux System Tuning (as of June 28 2018) provides guidance based on tuning work done on 2013




• in connected mode, at the time of the recommendations made on this wiki page in 2014, was known to have scaling issues, resource usage issues, and stability issues. 
  
  Again, please consult your network vendor for their recommendations on the IPoIB mode to use