Introduction
PowerVM’s SR-IOV technology allows a partition to have direct access to a shared physical adapter to provide low overhead, high performance networking. Previously, partition mobility operations were not allowed when a partition’s configuration included direct access to physical I/O adapters. As part of Power Systems firmware FW940 & HMC 9.1.940, with supporting AIX and IBM i releases, IBM is introduced as a Technology Preview a new PowerVM capability called Hybrid Network Virtualization (HNV). With Power Systems firmware FW940.10 & HMC9.1.941.0, with supporting AIX and IBM i releases, IBM introduced production ready HNV support. With Power Systems firmware FW950.00 & HMC 9.2.950.0, along with supporting Linux releases, IBM introduced Linux HNV support. HNV allows AIX, IBM i, and Linux partitions to take advantage of the efficiency and performance benefits of SR-IOV logical ports and participate in mobility operations such as active and inactive Live Partition Mobility (LPM) and Simplified Remote Restart (SRR). HNV is enabled by selecting a new Migratable option when an SR-IOV logical port is configured.The new Migratable option for Hybrid Network Virtualization (HNV) capability is provided as an unsupported Technology Preview only at this time and is not intended for production deployments. Introduction as a Technology Preview will allow users to gain early experience with this technology and provide feedback to IBM. Comments, questions, or issues are welcome via our PowerVM LinkedIn forum or in the comments section of this blog. The IBM development teams responsible for this technology will address questions and issues as time allows.
Overview
Previously partitions were not mobility capable if physical I/O devices were assigned to the partition. This included SR-IOV logical ports as they provide direct access to adapter Virtual Functions (VFs). For AIX and Linux partitions configured with physical devices, the physical devices could be manually removed from the partition prior to starting a mobility operation. For IBM i partitions, configuration of physical devices was not allowed if the partition was enabled for mobility.Hybrid Network Virtualization leverages existing technologies such as AIX Network Interface Backup (NIB) and IBM i Virtual IP Address (VIPA) as its foundation and introduces new automation for configuration and mobility operations to provide an integrated high-performance network adapter sharing capability with partition mobility.The approach is to create an active-backup configuration within a partition where the primary device is an SR-IOV logical port and the backup device is a virtual device such as a Virtual Ethernet adapter or virtual Network Interface Controller (vNIC). As the primary device, the SR-IOV logical port provides high performance low overhead network connectivity. During an LPM operation, or when the primary device cannot provide network connectivity, network traffic will flow through the backup virtual device. As mentioned previously, to migrate a partition the partition may only have virtual I/O. If the partition is configured with a Migratable SR-IOV logical port, the Hardware Management Console (HMC) will dynamically remove the SR-IOV logical port as part of the migration operation. This forces network traffic to flow through the virtual backup device. With the SR-IOV logical ports removed, the HMC is able to migrate the partition. Prior to migration, the HMC will provision SR-IOV logical ports on the destination system to replace the previously remove logical ports. Once the partition is on the destination system the HMC will dynamically add the provisioned logical ports to the partition where they will be integrated into the active-backup configuration (e.g. NIB or VIPA). Figure 1 provides an example active LPM flow for an AIX partition with a Migratable logical port. The various high-level phases are described below.Phase 1 is the normal configuration prior to the LPM. The green lines represent the high-performance primary path for the NIB through an SR-IOV logical port (VF) assigned to the partition. The red dashed lines represent the backup path. In this example the backup path is via a vNIC backup device. Phase 2 is the configuration prior to the actual move of the partition to the destination system. At this phase the HMC has dynamically removed the Migratable SR-IOV logical port. This forces the network traffic to flow over the backup vNIC device. With the Migratable SR-IOV logical port removed, the partition configuration only includes virtual devices. Phase 3 shows the AIX LPAR on the destination system with the vNIC backup device as the active path for network traffic.Once the partition is on the destination system, the HMC dynamically adds a Migratable SR-IOV logical port to the partition. The partition recognizes the Migratable logical port as belonging to a specific active-backup configuration and adds it as the primary device in the Etherchannal device associated with the backup vNIC device. The primary device then becomes the active device for network traffic as illustrated in phase 4.
Configuration
Configuration of HNV devices is a two-step process. The first step is to configure a Migratable logical port and its backup device at the HMC for a partition profile or a partition. The new devices become visible to the partition on activation of the partition profile or when the Migratable logical port is dynamically added to the partition. Once the devices are visible to the partition, OS specific configuration steps are required to complete the configuration (see AIX Configuration and IBM i Configuration below).
HMC Configuration
To indicate an SR-IOV logical port is to be a HNV device, a new Migratable option has been added to the HMC GUIs and CLIs. Note: A Migratable logical port must have its OS MAC Address Restrictions set to Allow All OS defined MAC addresses. Also, a Migratable logical port and its backup device cannot be configured with the same physical port.
HMC GUI Configuration
To configure a Migratable logical port on the HMC GUI the user uses the normal Add SR-IOV Logical Port GUI. In the Advanced Settings section there is a new check box for selecting the Migratable option. When the Migratable option is selected additional fields are displayed for selecting and configuring the backup device (see Figure 2 - Add SR-IOV Logical Port with Migratable Option). The options for a backup device are Virtual Ethernet Adapter and Virtual NIC Adapter (vNIC).
To configure a Virtual Ethernet Adapter as the backup device, select Virtual Ethernet Adapter and click on the Configure backup device. This will display an Attached Virtual Network panel where the desired Virtual Network can be selected (see Figure 3 - Virtual Ethernet Adapter Configuration).
If you select Virtual NIC Adapter as the backup device, a default configuration is displayed (see Figure 4- Virtual NIC (vNIC) Adapter Backup Device).
If a different Virtual NIC Adapter configuration is desired, click on the Configure backup device under the Backing device Type. This will displays a panel with options for selecting the physical port, Hosting Virtual I/O Server partition and Capacity value for the Virtual NIC Adapter backup device (see Figure 5- Virtual NIC (vNIC) Adapter Configuration).
Similar capabilities are available when configuring a Migratable SR-IOV logical port in a partition profile.
HMC CLI Configuration
The following provides various examples of how to use the HMC CLI for various HMC operations.
Dynamic Add of a Migratable Logical Port1. Use all defaults:
chhwres -r sriov -m fsp-tiamat –rsubtype logport –id 54 -o a -a "adapter_id=1,phys_port_id=1,logical_port_type=eth,migratable=1”
Note: HMC will select a SR-IOV physical port whose label matches the label of the physical port the Migratable logical port will be on.
2. Configure the backup vNIC backing device
chhwres -r sriov -m fsp-tiamat –rsubtype logport –id 54 -o a -a “adapter_id=1, phys_port_id=1, logical_port_type=eth,migratable=1,backup_device_type=vnic,backup_device_slot_num=2,backup_vnic_backing_device=sriov/tiamat-VIOS-VIOS1/5/2/0/0//100.0”
3. Use VETH as backup device
chhwres -r sriov -m fsp-tiamat –rsubtype logport –id 54 -o a -a “adapter_id=1, phys_port_id=0, logical_port_type=eth,migratable=1,backup_device_type=veth,backup_device_slot_num=9,backup_veth_vnetwork=VLAN1-ETHERNET0”
List Migratable Logical ports of an LPAR
lshwres -r sriov -m fsp-tiamat --rsubtype logport --level eth --filter "lpar_names=SRIOVMigrationLP"
config_id=0,lpar_name=SRIOVMigrationLP,lpar_id=54,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004001,logical_port_type=eth,drc_name=PHB 4097,location_code=U78D2.001.WZS0107-P1-C9-T2-S1, functional_state=1,phys_port_id=1,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=1.0,max_capacity=100.0, promisc_mode=0,mac_addr=8e17bbd0e100,curr_mac_addr=8e17bbd0e100,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=0, migratable=1,backup_device_slot_num=2,backup_device_type=vnic,backup_vnic_backing_device=sriov/tiamat-VIOS1/5/2/0/2.0/50/100.0,is_vnic_backing_device=0
config_id=1,lpar_name=SRIOVMigrationLP,lpar_id=54,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004002,logical_port_type=eth,drc_name=PHB 4098,location_code=U78D2.001.WZS0107-P1-C9-T1-S2, functional_state=1,phys_port_id=0,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=1.0,max_capacity=100.0, promisc_mode=0,mac_addr=8e17bf4ded01,curr_mac_addr=8e17bf4ded01,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=0, migratable=1,backup_device_slot_num=9,backup_device_type=veth,backup_veth_vnetwork=VLAN1-ETHERNET0,is_vnic_backing_device=0
Dynamic Remove/Modify of a Migratable Logical Port
- Syntax for a Migratable logical port is the same as a logical port without the Migratable option selected
- When a Migratable logical port is removed, its associated backup device is automatically removed too
- Dynamic Modify of a Migratable logical port will modify some backup device properties when the backup is a vNIC device. Properties modify for a vNIC backup device include the PVID and allowed VLAN list
- A backup device of a Migratable logical port cannot be removed unless the Migratable logical port is not configured. This can happen after LPM or SRR.
OS ConfigurationOnce a Migratable logical port and its backup are configured at the HMC and dynamically added to a partition or a partition is activated with a new Migratable logical port, OS level configuration can commence. Since Migratable logical ports leverages existing OS active-backup technologies, configuration in the OS is the same for a Migratable logical port and its backup as it is for other active-backup configurations. AIX ConfigurationWhen cfgmgr executes and detects a newly defined Migratable logical port an Etherchannel / IEEE 802.3ad Link Aggregation device is created with the Migratable logical port as the only Etherchannel Adapter and the backup device (virtual Ethernet or VNIC) as the backup_adapter. At this point the configuration of the Etherchannel device with Migratable logical port can proceed as it would for other Etherchannel devices (e.g. configure IP address).IBM i ConfigurationIBM i partitions use VIPA as the underlying technology for Migratable logical port active-backup configuration. To complete the Migratable logical port configuration the user must setup a VIPA configuration using the Migratable logical port and backup CMN resources.
Linux Configuration
Linux partitions use active-backup bonding as the underlying technology for Migratable logical port active-backup configuration. Once a Migratable logical port and its backup virtual devices are configured at the HMC and dynamically added to a Linux partition, or a Linux partition is activated with a new Migratable logical port, the rest of the configuration of the bonding device, such as configure IP address, can proceed as normal bonding device configuration.
Mobility Operations
One of the key advantages of the HNV approach is the integration of the dynamic remove and add of the Migratable logical port as part of active LPM, inactive LPM, or SRR operations.
When a partition mobility operation is initiated, the HMC determines the SR-IOV adapter physical port label of Migratable logical ports. The HMC uses the physical port label to identify candidate physical ports for a Migratable logical port. At this time manual mapping of the Migratable Logical Port is not supported.When a partition to be migrated contains Migratable logical ports, a new override option “Only migrate SR-IOV logical ports when possible” will be visible. Selecting it means: if HMC cannot find a mapping for a Migratable logical port, migration is still allowed. HMC will not recreate the Migratable logical port after the mobility operation. The user can manually recover the Migratable logical port after the mobility operation.
Live Partition Mobility (LPM) and Simplified Remote Restart (SRR) CLI examplesmigrlpar -m fsp-tiamat -t hmc-zzfp139 –id 54 -o [v | m] --migsriov 2rrstartlpar -m fsp-tiamat -t hmc-zzfp139 –id 54 -o [validate | restart] --migsriov 2Recover Migratable VFs (CLI only)With the –migsriov 2option, or other unexpected error while trying to recreate the logical port on the target system, the partition will end up having virtual adapters without primary logical port. In this case, you can manually recover the logical ports using the virtual adapter slot ID:chhwres -r sriov -m hmc-zzfp139 –-rsubtype logport –-id 54 -o rc -a “adapter_id=1,phys_port_id=0,logical_port_type=eth,capacity=10,max_capacity=20,backup_device_slot_num=3”Partition Template Impacts
- All Migratable Logical Ports will be captured in the partition template.
- A Migratable Logical Port’s configuration can be changed by editing the partition template.
- A Migratable Logical Port and/or its backup device can be configured during template deployment.
Technology Preview Minimum Requirements
- IBM POWER9 based Servers
- System firmware: FW940 or later.
- HMC version/release: HMC 9.1.940.0 or later.
- AIX Version 7.2 with the 7200-04 Technology Level or later
- IBM i 7.3 TR7 and IBM i 7.4 TR1
Production Ready Minimum Requirements
- IBM POWER9 based Servers
- System firmware: FW940.10 or later. For Linux, FW950.00 or later
- HMC version/release: HMC 9.1.941.0 or later. For Linux HMC 9.2.950.0 or later
- AIX Version 7.2 with the 7200-04 Technology Level and Service Pack 7200-04-02-2015
- IBM i 7.3 TR7 and IBM i 7.4 TR1
- VIOS 3.1.1.20
- RedHat Enterprise Linux 8.4
- SUSE Linux Enterprise Server 15 Service Pack 3 or later. Requires NetworkManager package and NetworkManager service to be enabled.
- Linux support requirements:
- Backup device limited to Virtual Ethernet.
- Supported adapter FCs EC2R/EC2S, EC2T/EC2U, EC3L/EC3M, EC66/EC67
Contacting the PowerVM TeamHave questions for the PowerVM team or want to learn more? Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions#PowerVM#powervmblog#powervmlpm