PowerVM

  • 1.  Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Mon March 01, 2021 10:22 PM
    I recently ran into a situation where I'm not entirely sure I understand 100% if I'm correct.  It is a configuration of a system with two VIOS servers in failover mode, with LPARs on three different VLANs contained in the server.  I have several questions:

    1. When creating a Shared Ethernet Adapter in a two VIOS failover setup, the mkvdev can also specify an IP address attribute (-attr netaddr=x.x.x.x) which is then associated with the SEA being configured. 
      1. Is this required?
      2. Is this for exclusive use of the control channel to ping the other VIOS to check on its availability? Or can it also be used for external administration/management communications?
      3. If used exclusively for the control channel, do I create another virtual adapter that connects to the SEA and give that adapter its own IP address?
      4. If external admin/mgmt is on its own VLAN (12 for example), is it enough to just do "mkvdev -vlan SEAentX -tagid 12", give it a different IP address and expect external hosts to be able to communicate with that VIOS? (SEA is on VLAN ID 999, in my example)
      5. What happens if you fail to specify an IP address on both VIOS?
    2. As stated above, the system will have LPARs on three different VLANs:
      1. Is it absolutely necessary to create separate virtual ethernet adapters for each VLAN with "mkvdev -vlan entX -tagid vlanid"?
      2. Is it instead possible to have all LPAR's virtual ethernet client adapters (across three different VLANs) to connect to a single virtual ethernet server adapter in the VIOS servers (VLAN 99, which no host on the network uses)?  Is this recommended?
    Thanks for anyone who can offer clarity!

    Carlo


  • 2.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup
    Best Answer

    Posted Tue March 02, 2021 03:22 AM
    1a) No
    1b) This address is the IP address of the external entity that would be used by VIOS to verify SEA access to outside world by ICMP ping. So this is not a local address (it does not get assigned to any interface on VIOS) and therefore one cannot use it for network traffic from/to VIOS.
    1c) N/A as answer to 1b) makes it so.
    1d) Yes, although configuring network access to VIOS by assigning IP address to SEA interface is usually not the best idea  - you might not be able to reconfigure SEA without disrupting communication from/to this address. This in turn means that for such action you'd have to login to VIOS from virtual console as an SSH session could get cut off by your own actions (SEA reconfiguration).
    1e) I am not sure which IP address you are referring to. If it is the one you write about in 1a), then in general nothing would happen, but this means each SEA would base its usability solely on the real interface link state and ha_mode attribute setting. Whether this is what you really want actually depends on you and your environment.
    2a) No. You may have multiple VLANs bridged by one virtual adapter. Limit is 20 - 1 specified in as PVID and 19 specified by additional VLANs set. Waring: VLAN tags for PVID gets stripped from network frames on egress from vswitch, so this might not be what you really want. In particular, having multiple briging virtual network interfaces when at least one of them have just PVID specified (no additional VLANs) is most likely not what you want as it won't get you all those VLANs bridged properly.
    2b) First, there are no "server vs client" differentiation for virtual network adapters. There are bridging and non-bridging ones, depending on "trunk-priority" value. Having said so, if you mean something like "configure SEA with just one bridging adapter that has VLAN 99 as PVID and VLANs X, Y and Z as additional VLANs and use this SEA to service traffic to/from other LPARs that use VLANs X, Y or Z" then yes, this will work. Is it recommended over having three separate bridging virtual network interfaces? Usually yes, because you get almost the same flexibility (additional VLANs set can be modified on-the-fly by DLPAR actions) while wasting less VLAN IDs - basically not more than one of all PVIDs in bridging adapters may be effectively used, as mentioned in 2a). Please remember about limits for a single virtual adaptera (no more than 19 additional VLANs), one SEA (no more than 16 bridging virtual adapters) and one LPAR (no more than 256 virtual ethernet adapters.

    ------------------------------
    Lech Szychowski
    ------------------------------



  • 3.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Wed March 03, 2021 03:15 AM
    Thanks for taking the time to type out that detailed response, Lech!  If I may, I'd like to keep my follow up questions answerable by yes or no.

    > 1a) When creating a Shared Ethernet Adapter in a two VIOS failover setup, the mkvdev can also specify an IP address attribute (-attr netaddr=x.x.x.x) which is then associated with the SEA being configured. Is this required?
    > No

    So, basically, I can have an SEA failover setup between two VIO servers without any IP addresses?

    > 1b) Is this for exclusive use of the control channel to ping the other VIOS to check on its availability? Or can it also be used for external administration/management communications?
    > This address is the IP address of the external entity that would be used by VIOS to verify SEA access to outside world by ICMP ping. So this is not a local address (it does not get assigned to any interface on VIOS) and therefore one cannot use it for network traffic from/to VIOS.

    So although the "-attr netaddr" attribute is specified in the same mkvdev command that creates the SEA itself, that IP address is not directly associated to the SEA itself, correct?

    Also, this IP address is not something I can ssh into from the external network?

    > 1c) If used exclusively for the control channel, do I create another virtual adapter that connects to the SEA and give that adapter its own IP address?
    > N/A as answer to 1b) makes it so.

    Got it.

    > 1d) If external admin/mgmt is on its own VLAN (12 for example), is it enough to just do "mkvdev -vlan SEAentX -tagid 12", give it a different IP address and expect external hosts to be able to communicate with that VIOS? (SEA is on VLAN ID 999, in my example)
    > Yes, although configuring network access to VIOS by assigning IP address to SEA interface is usually not the best idea - you might not be able to reconfigure SEA without disrupting communication from/to this address. This in turn means that for such action you'd have to login to VIOS from virtual console as an SSH session could get cut off by your own actions (SEA reconfiguration).

    This is a bit of a gray area for me because I've seen in several documents (e.g., IBM PowerVM Virtualization Introduction and Configuration, for one) saying that the IP address should be configured on the SEA adapter itself, but your explanation makes perfect sense because of the availability implications.  Guess I have to do a bit more reading on this specific point.

    1e) What happens if you fail to specify an IP address on both VIOS?
    > I am not sure which IP address you are referring to. If it is the one you write about in 1a), then in general nothing would happen, but this means each SEA would base its usability solely on the real interface link state and ha_mode attribute setting. Whether this is what you really want actually depends on you and your environment.

    Yes, you understood it correctly.  I guess my question now should be, is it proper to configure an SEA failover between VIO servers without configuring these IP addresses on the two VIOS?

    > 2a) As stated above, the system will have LPARs on three different VLANs. Is it absolutely necessary to create separate virtual ethernet adapters for each VLAN with "mkvdev -vlan entX -tagid vlanid"?
    > No. You may have multiple VLANs bridged by one virtual adapter. Limit is 20 - 1 specified in as PVID and 19 specified by additional VLANs set. Waring: VLAN tags for PVID gets stripped from network frames on egress from vswitch, so this might not be what you really want. In particular, having multiple briging virtual network interfaces when at least one of them have just PVID specified (no additional VLANs) is most likely not what you want as it won't get you all those VLANs bridged properly.

    Makes sense.

    > 2b) Is it instead possible to have all LPAR's virtual ethernet client adapters (across three different VLANs) to connect to a single virtual ethernet server adapter in the VIOS servers (VLAN 99, which no host on the network uses)? Is this recommended?
    > First, there are no "server vs client" differentiation for virtual network adapters. There are bridging and non-bridging ones, depending on "trunk-priority" value. Having said so, if you mean something like "configure SEA with just one bridging adapter that has VLAN 99 as PVID and VLANs X, Y and Z as additional VLANs and use this SEA to service traffic to/from other LPARs that use VLANs X, Y or Z" then yes, this will work. Is it recommended over having three separate bridging virtual network interfaces? Usually yes, because you get almost the same flexibility (additional VLANs set can be modified on-the-fly by DLPAR actions) while wasting less VLAN IDs - basically not more than one of all PVIDs in bridging adapters may be effectively used, as mentioned in 2a). Please remember about limits for a single virtual adaptera (no more than 19 additional VLANs), one SEA (no more than 16 bridging virtual adapters) and one LPAR (no more than 256 virtual ethernet adapters.

    You got it exactly right (the part which I italicized in your reply).

    Again, awesome reply!  Thanks!

    Carlo Castillo


  • 4.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Wed March 03, 2021 04:43 AM
    > So, basically, I can have an SEA failover setup between two VIO servers without any IP addresses?

    Correct. Cooperating/redundant SEAs use control channel virtual adapter to communicate. This communication does not need any IP setup.

    BTW: since some time (read: in newer software versions) it is not necessary to have a virtual Eth adapter specified for control channel any more. See Shared Ethernet Adapter (SEA) Simplification : No dedicated control channel adapter document for details.

    > So although the "-attr netaddr" attribute is specified in the same mkvdev command that creates the SEA itself, that IP address is not directly associated to the SEA itself, correct?

    Correct. If it were to be assigned to an interface, it would be also necessary to specify netmask...

    > Also, this IP address is not something I can ssh into from the external network?

    This address is assigned to another device so it depends on what this device is and how it is configured :)

    > This is a bit of a gray area for me because I've seen in several documents (e.g., IBM PowerVM Virtualization Introduction and Configuration, for one) saying that the IP address should be configured on the SEA adapter itself,

    Well, as a matter of fact whenever I can I configure networking in VIOS system on real physical interfaces. This way VIOS systems are accessible via SSH no matter if and how all SEAs are configured there. If there are no physical interfeaces available for this, I configure network acces on a separate virtual Eth, not the SEA, for the reasons described earlier.

    > is it proper to configure an SEA failover between VIO servers without configuring these IP addresses on the two VIOS?

    Yes. Bridging network traffic by SEA is completely independent of the network configuration (IP addresses on interfaces, routing) in VIOS.

    > configure SEA with just one bridging adapter that has VLAN 99 as PVID and VLANs X, Y and Z as additional VLANs and use this SEA to service traffic to/from other LPARs that use VLANs X, Y or Z

    One warning here: if you plan on using SEAs in load sharing mode rather than failover mode, be aware of the way this load sharing works.
    The way it does is SEAs split the set of briding interfaces among  them - one SEA takes care of bridging traffic via half of the adapters, the other one bridges the other half. Therefore you should not group highly used VLANs in one bridging adapter, because this way only one SEAs will always service all these VLANs.
    I usually configure SEAs in load sharing mode with at least two bridging adapters (so that load sharing can take place) and pay attention to proper assignment of additional VLANs to these adapters according to traffic characteristics in these VLANs.
    And no, you have no control of how the set of interfaces is split. So if you have more than 2 interfaces, it is more or less a lottery (well, not quite, but no real control).

    So the example looks more like this:
    - configure SEA with two bridging adapters
    - have first adapter have VLAN P as PVID and VLANs X and Y as additional VLANs
    - have first adapter have VLAN A as PVID and VLANs B and C as additional VLANs
    - set SEA to load sharing mode (ha_mode=sharing)
    Then this pair of SEAs will service traffic to/from other LPARs that use VLANs X, Y, A and B this way that in normal situation (both VIOSes up and running, no network/interfae problems) one SEA service VLANs X and Y, and the other one takes care of VLANs X and Y.

    ------------------------------
    Lech Szychowski
    ------------------------------



  • 5.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Wed March 03, 2021 05:29 AM
    On Wed, Mar 03, 2021 at 09:42:55AM +0000, Lech Szychowski via IBM Community wrote:
    > Well, as a matter of fact whenever I can I configure networking in
    > VIOS system on real physical interfaces. This way VIOS systems are
    > accessible via SSH no matter if and how all SEAs are configured
    > there. If there are no physical interfeaces available for this, I
    > configure network acces on a separate virtual Eth, not the SEA, for
    > the reasons described earlier.

    Dedicated adapters for VIO management IP are easier than ever with the
    newer 4 port cards which have 2 10G and 2 1G ports.

    I recall in one of the redbooks where they recommend against putting
    an IP address on the SEA. Remember the SEA and the physical adapters
    it uses are in promiscuous mode, so all packets transiting the SEA
    bridge must be scanned to see if they match that SEA IP using your
    POWER CPU in the software layer TCP/IP stack. That is very CPU
    intensive and scales with adapter speed. Typically switches filter
    packets by MAC for you.

    If you can't dedicate physical adapter ports to VIO management, then I
    recommend creating a client virtual Ethernet adapter on the VIO (ie:
    no trunk or bridge) attached to a VLAN the VIO servers bridge. Thus
    VIO is a client of itself, and fails over with the SEA bridge
    connecting the vswitch to the external network.

    Clearly dedicated ports are preferred.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    http://adamssystems.nl/




  • 6.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Tue March 09, 2021 07:32 PM
    Thanks Russell! Cheers!

    ------------------------------
    Carlo Castillo
    ------------------------------



  • 7.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Tue March 09, 2021 07:31 PM
    Thanks again for your insightful replies to my questions, Lech!  They've been extremely helpful!

    > So, basically, I can have an SEA failover setup between two VIO servers without any IP addresses?

    Correct. Cooperating/redundant SEAs use control channel virtual adapter to communicate. This communication does not need any IP setup.

    One last question:  Since I configured an IP address on my SEA, is it safe to remove the IP address while the VIO servers are already running?  How would I go about doing this?

    Thanks again!

    ------------------------------
    Carlo Castillo
    ------------------------------



  • 8.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Wed March 10, 2021 04:24 AM
    In general there is nothing special here, you should take the same steps as with removing any IP address from any network interface in an AIX/VIOS system.
    If the address you are to remove is the one that you normally use to access this system, you have to work out how to access this system in a different way, so that you could remove the address without cutting  yourself off from this system. Using system console is the usual solution for this problem.
    Of course you should also make sure that the address you're about to remove is not used in any solutions you might have configured in your environment (monitoring, BigFix/Ansible etc).

    ------------------------------
    Lech Szychowski
    ------------------------------



  • 9.  RE: Shared Ethernet Adapter Failover configuration in dual VIOS setup

    Posted Wed March 10, 2021 08:36 AM