Networking in  IBM Virtual Dev and Test for z/OS 

With the announcement of IBM Virtual Dev and Test for z/OS (ZVDT) comes the ability to leverage Linux on Z as a hypervisor to instantiate z/OS Instances (guests). One of the prerequisites for deploying z/OS Instances using ZVDT is to configure compatible networking resources on Linux so there is connectivity to the z/OS Instances.  Future editions of ZVDT may include the necessary tooling to setup the networking interfaces automatically, however, until such time this is a manual procedure. ZVDT currently makes use of MacVTap virtual adapters. 

The following document provides a good outline of using MacVTap => Using the MacVTap driver

For ZVDT, there are a few differences versus what is outlined in the "Using the MacVTap driver" link above. ZVDT, for example, will be instantiating z/OS Guests (not other Linux Guests). ZVDT also does not make use of libvirt or XML stanzas for defining interfaces. The following guidelines should help with getting the MacVTap interfaces setup correctly for ZVDT.

The recommendation for ZVDT is to have at least two OSA CHPID's. One to use for the hosting Linux system networking, which should already be setup and configured, and the other to use for z/OS Instances. 

Finding already configured devices:

To determine what Interfaces are already configured, use the Linux znetconf -c command, for example:
$ sudo znetconf -c
Device IDs                 Type    Card Type      CHPID Drv. Name             State  
-------------------------------------------------------------------------------------
0.0.1500,0.0.1501,0.0.1502 1731/01 OSD_1000          04 qeth enc1500          online 

The above example shows that there is one interface (enc1500 on OSA CHPID 04) which, for the purposes of this document, will be assumed to be primary Linux interface. Another interface using devices on a different CHPID will be needed. 

Finding un-configured devices:

To find un-configured devices,  use the Linux znetconf -u command, for example:

$ znetconf -u
Scanning for network devices...
Device IDs                 Type    Card Type      CHPID Drv. 
------------------------------------------------------------
0.0.ff43,0.0.ff44,0.0.ff45 1731/05 HiperSockets      03 qeth 
0.0.ff46,0.0.ff47,0.0.ff48 1731/05 HiperSockets      03 qeth 
0.0.ff49,0.0.ff4a,0.0.ff4b 1731/05 HiperSockets      03 qeth 

0.0.1503,0.0.1504,0.0.1505 1731/01 OSA (QDIO)        04 qeth
0.0.7203,0.0.7204,0.0.7205 1731/01 OSA (QDIO)        e0 qeth 
0.0.7209,0.0.720a,0.0.720b 1731/01 OSA (QDIO)        e0 qeth 
0.0.7220,0.0.7221,0.0.7222 1731/01 OSA (QDIO)        e2 qeth 
0.0.7223,0.0.7224,0.0.7225 1731/01 OSA (QDIO)        e2 qeth 

The above display shows several un-configured devices on CHPID's 03, 04, e0, and e2. It is important to note that MacVTap adapters do not support being defined on HiperSockets and therefore the devices listed as type HiperSockets (CHPID 03 in above example) cannot be used. The devices listed under CHPID 04, in the above example, also should not be used as they are on the same OSA CHPID as primary Linux interface (shown in znetconf -c example above). The OSA (QDIO) devices listed for CHPID e0 or e2 are good candidates to use in this case.  

If you find there are no devices listed from the znetconf -u command then it is possible that all other devices are subject to cio_ignore blacklist, or it is possible there really are no other devices in your Linux configuration available for use. In the first case, you may be able to release devices from the cio_ignore blacklist by Linux command cio_ignore -r 0.0.xxxx, however, you will need to know which specific devices you can use. Reference your hardware configuration, or speak with those in your organization that provide hardware configuration to obtain this information or request changes so there are available devices to use on a different CHPID than the primary interface.

In addition to cio_ignore, you may also need to configure the devices online prior to the devices being listed by the znetconf -u command or being available for configuration with command znetconf -c. For the supported Linux distributions, which as of this writing are Red Hat and Ubuntu, you can use the chzdev command enable devices, for example, sudo chzdev -e 0.0.7203. This would enable device 0.0.7203 (along with the 2 others in the range) and ensure they becomes part of the persistent configuration.  Other commands that may be useful are lszdev, cio_ignore -l, cio_ignore -L. Once the devices are available to Linux then configure them as network interfaces.

Defining the interfaces:

To add a new interface use znetconf -a, for example:
sudo znetconf -a 0.0.7203
The above command should add a Linux adapter named enc7203

Next define a Virtual MacVTap Adapter with any name you like. The below example is using myvtap as the name, and it is linked to base device enc7203 that was just defined:
sudo ip link add link enc7203 myvtap type macvtap mode bridge

Set the new MacVTap adapter to UP status:
sudo ip link set myvtap up

It’s important to note that there will not be any IP addresses assigned to the MacVTap interface or base interface when using public IP addresses, the IP address will be injected to z/OS TCP/IP configuration by ZVDT tooling. MacVTap does have a limitation where communication from the Linux Host to z/OS will not work when using the same subnet between the guests and the hosting Linux system, however the above examples get around this limitation by using separate network interfaces for Linux and z/OS. You may also need to take steps to setup the networking configuration so the interfaces are automatically configured across Linux reboots, this is different than chzdev mentioned above which simply ensures the devices are online.

Now that the virtual adapter is ready (after setting the status to UP), z/OS can be deployed with ZVDT tooling. The above steps are hopefully helpful in getting the networking prerequisites satisfied. The ZVDT Web UI will prompt for Default Route, MacVTap adapters to use as well as a range of IP Addresses that can be used to deploy multiple z/OS Instances.

OTHER CONSIDERATIONS

Arp Flux
Arp Flux can be a problem if you define a MacVTap adapter on the same OSA CHPID as the primary Linux host interface, which is why this is not recommended. Arp Flux will manifest itself as intermittent periods of network delays/timeouts. 

TCP/IP Segmentation Offload settings:
When using ZVDT it is best to disable the offloading parameters for the network interface(s) that will be used by z/OS Instances, to avoid networking performance issues. The Linux ethtool command can be used to change these settings, for example:
sudo ethtool -K enc7203 tso off gro off gso off rx off tx off

VLAN Tagging:
Although the example above do not show interfaces that may be subject to VLAN tagging, these have been tested and function as expected. The Linux Interfaces are setup normally as you would with VLAN Tagging, for example if interface enc7203 was subject to vlan tagging, then after enc7203 was initially configured (znetconf -a 0.0.7203) you would add a new tagged interface with the correct VLAN, for example: sudo ip link add link enc7203 name enc7203.2231 type vlan id 2231
Then create your MacVTap interface normally but it would be linked with the interface defined with type VLAN :
sudo ip link add link enc7203.2231 myvtap type macvtap mode bridge

Private IP Addressing:
Private IP addressing can also be used for z/OS. This will mean that you can only connect to z/OS from the Linux Host where the z/OS Instance is running, perhaps with using C3270, or configuring a VPN server on the Linux Host. The only real difference with using Private IP addresses is you will need to assign the main Interface, from which MacVTap adapters are linked to, with the “gateway” address.

For example, if we use enc7203 with private IP addressing then we could assign 172.26.1.1 to the main interface to use as the gateway:
sudo ip addr add 172.26.1.1/24 dev enc7203

Then create your MacVTap interface, however, the MacVTap adapters should be created on a different set of devices, but could be on same OSA CHPID.
sudo ip link add link enc7206 myvtap type macvtap mode bridge

z/OS can now be provided a private IP Address like 172.26.1.2 with the default route configured as 172.26.1.1.  Connect to z/OS using the private IP either directly from the Linux host, or by setting up VPN.