PowerVM

EtherChannel MAC Swap For Certain Complex Network Environments Utilizing SR-IOV and vNIC Clients

By Christina Morel posted Wed January 20, 2021 11:39 AM

  

In certain complex network environments utilizing SR-IOV and vNIC clients, we may see duplicate packets when unicast traffic arrives on both the active and idle ports of a NIB (Network Interface Backup) EtherChannel comprised of virtual adapters.  The MAC swap feature was implemented in order to enable users to avoid duplicate packets if they have a relevant network configuration.   

Normally in a NIB EtherChannel setup, the backup adapter and the EtherChannel itself are assigned the MAC address of the primary adapter.  If the primary adapter has MAC X, then the backup adapter and the EtherChannel itself will also have MAC X.  

In the event of a failover from the primary to the backup adapter, the backup adapter becomes active and the primary adapter becomes idle, but as you can see, all adapters still have MAC X. 

 

When the MAC swap feature is used (mac_swap=enabled), the backup adapter retains it’s original MAC address instead of being assigned the MAC address of the primary adapter.  The EtherChannel itself will be assigned the MAC address of the primary adapter.  So for example, if the primary, active adapter has MAC X and the backup, idle adapter has MAC Y, the EtherChannel will have MAC X.   

When an EtherChannel failover occurs from primary to backup (primary is now the idle channel and backup is now the active channel),  the original MAC address of the primary is assigned to the backup adapter, and the original MAC address of the backup is assigned to the primary adapter.  So now, the primary, idle adapter will have MAC Y and the backup, active adapter will have MAC X.  The EtherChannel itself retains MAC X.  

 
Duplicate Packet Example 

Now we will look at a complex example configuration to see how duplicate packets could be a problem for a NIB EtherChannel comprised of virtual adapters receiving unicast traffic.  In this configuration, we can assume that originally, LPAR2’s EtherChannel had an active primary port and an idle backup port.  Then at some point, a failover happened making the backup port active and the primary port idle.   

By default, the EtherChannel driver allows an idle virtual adapter to receive unicast traffic. This behavior exists in order to make sure packets are received by the EtherChannel in cases such as when the primary and backup adapters are on different vSwitches and a failover has occurred.  

 
Scenario 1: Traffic from LPAR 1 to LPAR2 

In the diagram below, you can see that when unicast traffic goes from LPAR1 to LPAR2, PHYP vSwitch0 will send the packets to the idle adapter of LPAR2’s EtherChannel.  If the driver didn’t allow the idle adapter to receive those packets, they would be lost. Because of this behavior, as we will discuss in scenario 2, in configurations such as our example, when LPAR3 sends unicast traffic to LPAR2, duplicate packets become a problem because both the active and the idle adapter receive packets.  

 


Scenario 2: Traffic from LPAR3 to LPAR2 with mac_swap Disabled
 

Let’s look at this scenario involving LPAR3 and LPAR2 to see why MAC swap mode is needed.  LPAR3 is sending unicast traffic to LPAR2.  Both the primary and backup adapters under the EtherChannel in LPAR2 are assigned the MAC address C.  As was mentioned, a failover occurred at some point previously, so the backup adapter is active and the primary adapter is idle.  We would expect traffic to go from LPAR3, through VIOS3 and on to the SR-IOV vSwitch 1.  From there, it would go  through the external switch in the middle into SR-IOV vSwitch 2, through VIOS2, and then up to the active backup port of LPAR2’s EtherChannel.  However, because the SR-IOV vSwitch 1 also sends the packet to the promiscuous port where it is picked up by the SEA in VIOS1, the traffic will go on to PHYP vSwitch 0.  PHYP vSwitch0 already knows to send traffic to MAC C and so the traffic will then go to LPAR2’s idle primary adapter, as well. 

 


Scenario 3: Traffic from LPAR3 to LPAR2 with mac_swap Enabled
 

Now let’s look at the scenario with the MAC swap feature turned on (mac_swap=enabled). The EtherChannel’s primary adapter and backup adapter have their own MAC addresses.  Because we know there was a failover previously, the backup adapter is active and it will have been assigned MAC C which is the same MAC that the EtherChannel itself is assigned.  The primary, idle adapter will have MAC D.  Now, when unicast traffic comes from LPAR3, when it gets to SR-IOV vSwitch 1, it will only go through the external switch and on to SR-IOV vSwitch 2.  It will also go on the promiscuous port through the SEA on VIOS2.  However, when it gets to PHYP vSwitch 0, it will drop the packets because it knows it sends traffic to MAC C, not MAC D.  Thus, the duplicate packet issue is avoided.  

 


Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions

0 comments
56 views

Permalink