Original Message:
Sent: Tue April 04, 2023 11:47 AM
From: Tsvetan Marinov
Subject: How to tell ethernet line speed on IBM i?
One caveat that is really important to mention is the Server memory, reserved for firmware to run the SR-IOV vNIC failover setup.
And think that's quite a lot, depending on the number of LPAR's that will have the vNIC failover setup.
In my example, I have 11 LPAR's, all using vNIC failover with 4 vNIC's for each LPAR(1 primary, 4 backup) and this takes 35GB of memory, reserved for the firmware, which is quite a large chunk of memory required. I think IBM should come up some proper sizing tool/software to calculate the memory overhead, based on the server configuration, number of LPAR's, number of vNIC adapters etc.
------------------------------
Tsvetan Marinov
Original Message:
Sent: Tue April 04, 2023 11:31 AM
From: Vincent Greene
Subject: How to tell ethernet line speed on IBM i?
It does get confusing. You get used to an adapter being assigned to an LPAR, and then SR-IOV goes and changes the whole pardigm.
The hypervisor owns the adapter, so you can assign subcapacity of the same port to multiple lpars. Even when talking about vNIC, which requires a VIOS lpar for management, there is no association between a particular VIOS and any given port. The same port can be used for two separate vNICs, each served by a different VIOS.
A picture may help. Left to right: Black lines are physical 10G connections to switch. All lines coming out of the right side of the ports represent a SR-IOV logical port or vNIC. The orange lines to the VIOS are SR-IOV ports used to create the VIOS management interface and shared ethernet (for migration). The lines to the vNIC boxes connected to the lpars show the vNIC backing device assignments. Solid line indicates the primary connection, Dashed line shows the backup connection. Color of these lines represents which VIOS manages that backing device. On the right side of the lpars is the virtual interfaces, and virtual switch that makes up the Shared Ethernet configuration, including the virtual switch.
Once migrated from the current SEA config to vNIC, there would be no need for the SEA part to the right of the lpars.
If you follow the lines from each lpar, you can see how each lpar has redundancy at the adapter, switch, and VIOS level (color).
------------------------------
Vincent Greene
IT Consultant
Technology Expert labs
IBM
Vincent.Greene@ibm.com
The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Original Message:
Sent: Mon April 03, 2023 07:30 AM
From: Robert Berendt
Subject: How to tell ethernet line speed on IBM i?
Thank you. I'm still a little confused. Is this two adapters with two ports each for each vios lpar? Or one adapter with two ports for each VIOS adapter?
BTW, guy down the street was changing out a switch just last week. His BP said that he should be safe to do one VIOS lpar at a time. Just unplug the cable from the old switch and put it on the new switch. What a mess. Had to replace his backplane, etc in his 9009-41A. Long night for him.
------------------------------
Robert Berendt IBMChampion
Original Message:
Sent: Fri March 31, 2023 06:58 PM
From: Vincent Greene
Subject: How to tell ethernet line speed on IBM i?
vNIC does not require any more hardware than SR-IOV. You can think of vNIC as essentially SR-IOV virtualized to add Live Partition Mobility and automatic failover support.
SR-IOV is the hardware functionality that allows the hypervisor to virtualize a physical port into logical ports that are assigned to lpars where they behave in all respects that matter as dedicated hardware for that lpar. SR-IOV alone does not require VIOS at all. The number of logical ports that can be supported by an SR-IOV adapter varies based on several factors, but for a 10Gb connection on a EC2U adapter, its way more than the 9 logical ports you would need in the described configuration.
For your two adapters, a basic configuration would be to
- assign both adapters to shared mode
- create 1 logical port per physical port (2 per adapter, 2 per switch) and assign two to each VIOS. Each VIOS would get one per switch from a alternate cards so each VIOS uses a port from each of the two cards. On the VIOS, these would be configured into a redundant pair. The resulting link aggregation adapter would be used as the bridge device for your Shared Ethernet adapter and also as the device for your VIOS management interface. This would be plain SR-IOV - not vNIC - because it is for the VIOS server. It would need to be configured in promiscuous mode to be used for the SEA. I'm talking about SEA here primarily to support migrating lpars one at a time. If you go with full vNIC for lpars from scratch you don't need an SEA.
- For each LPAR, create a vNIC with two or more backing devices spread evenly across your physical ports. Best practice would be to evenly split your primary backing devices across all your physical ports so normal state is balanced across ports, switches, and VIOS. Secondary backing devices would be used in a failoover situation. Best practice on those would be to use an alternate card, switch, and VIOS so a card or switch failure is handled. You can define more than two backing devices if you like.
Replace any VIPA configurations with vNIC and rely on the automatic failover to keep your addresses up. VIPA does not play well with vNIC or SEA because virtual adapters never register as down preventing VIPA from switching as it should in case of failure.
As I mentioned before, its possible to do one VIOS at a time and maintain a working SEA while transitioning, but there are a bunch of things to consider that depend on your specific network configuration, so I'm not even going to try to hit all the possibilities.
There are slot considerations for the adapters on P9 and below that vary based on specific power models and adapters. You'll want to pay attention to those, although in my experience, systems are usually shipped with cards in all the right places.
My blog https://blog.vios4i.com has an article on SR-IOV and vNIC that has more information and links to other sources for more information.
------------------------------
Vincent Greene
IT Consultant
Technology Expert labs
IBM
Vincent.Greene@ibm.com
The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Original Message:
Sent: Fri March 31, 2023 07:47 AM
From: Robert Berendt
Subject: How to tell ethernet line speed on IBM i?
Does vNIC require any more hardware than SR-IOV?
Let's say I have 7 partitions of IBM i, one partition of AIX and two VIOS partitions serving up ethernet, FC for SAN disk and FC for VTL.
Each VIOS adapter has one of these:
Description:PCIe3 2 PORT 25/10 Gb NIC&ROCE SFP28 ADAPTER
Location Code:U78DB.ND0.WZS01WR-P0-C8
Feature Codes:58FB, EC2U, EC2T
On each card one port goes to one switch and the other port goes to another switch.
------------------------------
Robert Berendt IBMChampion
Original Message:
Sent: Mon March 27, 2023 02:22 PM
From: Vincent Greene
Subject: How to tell ethernet line speed on IBM i?
The key is that in the VIOS-SEA configuration you have, there is no direct connection to the physical adapter(s), and there are usually multiple adapters to in a failover configuration between multiple VIOS, and very often aggregated adapters for a multiple of 10G actual bandwidth at the network switches.
What you have on the LPAR is a virtual adapter connected to a virtual switch in the Power Hypervisor. The VIOS has physical adapters bridged to virtual adapters which are connected to the same switch. Essentially it is a complete virtual network inside the hypervisor.
When IBM first developed virtual network adapters, they were most often used to provide fast connectivity between lpars on the same physical server. 1G was as fast as you can go on a physical adapter, so that is what the line speed shows. With the advent of line aggregation and faster NICs, that 1G started looking a bit anemic. The truth is the speed of that virtual network is limited by processor and memory on the LPAR itself, on the VIOS, and on the buffer sizes everywhere in the stack, so that, in the absence of specific tuning, that SEA is going to run at a speed closer to 1G than 10G, and you are never going to know because practically nobody ever pushes their networks as hard as they think they do.
In my opinion, Tsvetan hit the right answer with the suggestion to use vNIC with failover. The NICs you have are very nice SR-IOV capable adapters that will let you share ports with VIOS (vNIC) or without VIOS (SR-IOV logical ports) based on a hardware standard. vNIC is a best of both worlds -- eliminates bottlenecks in the data path, while still providing redundancy across multiple adapters.
You may have a complex path to follow to get to that point if you only have a couple of those cards providing VIOS management access and also acting as a network bridge for the virtual ethernet. It is probably possible with close to zero downtime if you already have a redundant shared ethernet configuration.
The trick would be to remove the dedicated EC2U from the backup VIOS and switch it to SR-IOV mode. You'd need to configure an SR-IOV logical port on the removed adapter (promiscuous mode) and assign it to the backup VIOS as its new management interface and SEA bridge device. Once you've confirmed the new logical port is working, you switch that VIOS to primary for the SEA and repeat the process for the other VIOS.
Once both adapters are in SR-IOV mode, you can create your vNIC and move each IBM i IP address to the new CMN resource one at a time. The only downtime would be during the switch to a new CMN resource on the client LPARs.
------------------------------
Vincent Greene
IT Consultant
Technology Services
IBM
Vincent.Greene@ibm.com
The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Original Message:
Sent: Fri March 24, 2023 10:05 AM
From: Robert Berendt
Subject: How to tell ethernet line speed on IBM i?
I get the impression that IBM couldn't ever get the "Current line speed" to show a valid value so they just stuffed it to match "Line speed". https://www.ibm.com/support/pages/apar/MA43080
------------------------------
Robert Berendt IBMChampion
Original Message:
Sent: Fri March 24, 2023 09:39 AM
From: Robert Berendt
Subject: How to tell ethernet line speed on IBM i?
Let's say you configured your line description with
CRTLINETH LIND(LANLIN) RSRCNAME(CMN03) ONLINE(*NO) ETHSTD(*ETHV2)
and WRKLIND, 5, shows:
Ethernet standard . . . . . . . . : *ETHV2
Line speed . . . . . . . . . . . . : *AUTO
Current line speed . . . . . . . . : *AUTO
Duplex . . . . . . . . . . . . . . : *AUTO
Current duplex . . . . . . . . . . : *AUTO
How do you tell what it is currently negotiated as?
What is the point of "Current line speed" vs "Line speed"?
New lpar, not in production yet. Open to changes on this.
------------------------------
Robert Berendt IBMChampion
------------------------------