Primary Storage

 View Only

Mellanox SX1710 40G switch MLAG configuration for ESS

By Archive User posted Thu July 12, 2018 08:37 AM

  
When we have 2 Mellanox 40G switches, we can use MLAG to bond ports between swithes, with server connected to these ports having bonding settings, the 40G network can have High Availability.

Switch front view:
switcher front view

Switch back view:
switcher back view

1. First, connect the serial console port of the swithes, finish the initialization process, give the switch management port an IP address, and then we can use the IP address to access the switch for configuration steps.

The default password for SX1710 switch is "User:admin; Password:admin"

2. Login to the management console of both swithes, check the version:

# show version

The Mellanox Switch needs to be at least firmware 3.3.5000 or later.

3. Open LACP and MLAG protocol

# en
# configure terminal
# lacp
# no spanning-tree
# ip routing
# protocol mlag
# dcb priority-flow-control enable force

4. Configure IPL link:
( We need connect 35 to 35, 36 to 36 port between two switches )

# interface port-channel 1
interface port-channel 1 ) # exit

# interface ethernet 1/35 channel-group 1 mode active
# interface ethernet 1/36 channel-group 1 mode active

# vlan 4000
vlan 4000) # exit

# interface vlan 4000
interface vlan 4000 ) # exit

# interface port-channel 1 ipl 1
# interface port-channel 1 dcb priority-flow-control mode on force

5. Configure IPL address and point each other as neighbor:
( These 2 ip addresses is not visible outside, it could be any IP address and subnet that is not in use in the network.)

On swither 1:

# interface vlan 4000
interface vlan 4000 ) # ip address 10.4.1.10 /24
interface vlan 4000 ) # ipl 1 peer-address 10.4.1.11
interface vlan 4000 ) # exit

On swither 2:
# interface vlan 4000 on switch 2
interface vlan 4000 ) # ip address 10.4.1.11 /24
interface vlan 4000 ) # ipl 1 peer-address 10.4.1.10
interface vlan 4000 ) # exit

6. Configure MLAG-VIP, MLAG use this to exchange keepalive message in Management network which management port connected to.
(The IP address should be the IP address assigned in the management network.)

# mlag-vip gpfs.net ip 192.168.*.* /22 force

7. Open Mlag globally:

# no mlag shutdown

8. Configure MLAG ports:

In below case, we have 3 servers connected to these two switches, each server have four 40G ports, 2 connected to switch one, 2 connected to switch two.

# interface mlag-port-channel 1-3
# interface mlag-port-channel 1-3) # exit

# interface ethernet 1/1 mlag-channel-group 1 mode active
# interface ethernet 1/2 mlag-channel-group 1 mode active

# interface ethernet 1/3 mlag-channel-group 2 mode active
# interface ethernet 1/4 mlag-channel-group 2 mode active

# interface ethernet 1/5 mlag-channel-group 3 mode active
# interface ethernet 1/6 mlag-channel-group 3 mode active

( mlag-channel-group same as mlag-port-channel)

9. Open MLAG interface

# interface mlag-port-channel 1-3 no shutdown
# interface ethernet 1/1-1/34 no shutdown

10.Change the MTU:
( server side bonding configuration should use same MTU )

# interface mlag-port-channel 1-3
interface mlag-port-channel 1-3) # mtu 9000 force
interface mlag-port-channel 1-3) # exit

11. Verify:

# show mlag
# show interfaces mlag-port-channel summary

12. Enable Flow control:

# interface mlag-port-channel 1-3 flowcontrol receive on force
# interface mlag-port-channel 1-3 flowcontrol send on force

On server side, we also need to check if flow control enabled, as below command:

ethtool -a

#ESS
#PrimaryStorage
#ElasticStorageServer
#Softwaredefinedstorage
#SpectrumScaleGNR
#Storage
#StorageAreaNetworks
#NetworkHA
#GPFS
0 comments
5 views

Permalink