Power

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

Hello
This VIRTUALIP behavior looks to me working as designed. The IBM i LPAR does not have any information about the external network setup, and its virtual ethernet adapters did not fail.

I believe that it is time to move from your current setup to a dual VIOS with SEA failover one. However, it is not not clear for me if you already run a dual VIOS environment or only one. Could you clarify please?

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

Hello:

The installation is a tiny one with 3 IBMi LPAR, so we have only one VIOS, with virtual disks (no NPIV). Reinstalling all the system with a Dual VIOS is not an option right now. Is there any chance to tell VIOS if the interface goes offline, the virtual Switch algo goes offline, and, with that solution, IBM i LPAR can switch to the other intrface using *VIRTUALIP?

Thank you in advance.

Best regards,

Hello
This VIRTUALIP behavior looks to me working as designed. The IBM i LPAR does not have any information about the external network setup, and its virtual ethernet adapters did not fail.

I believe that it is time to move from your current setup to a dual VIOS with SEA failover one. However, it is not not clear for me if you already run a dual VIOS environment or only one. Could you clarify please?

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Yes, several LPARs can share an SR-IOV capable ethernet adapter without the need of a VIOS. The sharing functionality of such an adapter is similar to the one we had in the past with IVE (Integrated Virtual Adpaters) if you are more familiar with this term.

However, not all ethernet adapters are SR-IOV capable, and installing those which support it depends on the server type.

So, to make further progress on your need, more information is needed.

What is the type/model of the server? Maybe this is a Power10 S1022 9105-22A? Is this another one?
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)?
Do you have empty slots in the server (or expansion units if any)?
And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Maybe a quick schema would help understanding your setup.

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Hello:

Thank you for your fast response. I'll try to answer the questions you sent me:

What is the type/model of the server? Power10 S1014 model 9105-41B
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? We have 2 cards 1 Gigabit Ethernet (UTP) 4 Port Adapter PCIE-4x/Short model 5899, 5260, one in slot U78DB.ND0.WZS03NR-P0-C10 and the other one in slot U78DB.ND0.WZS03NR-P0-C9

How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)? Yes, we have 2 cards with 4 port each. The first port on the first adapter is the VIOS en0 adapter communication, the rest of ports are SEA (one SEA per phisical port and adapter).

Do you have empty slots in the server (or expansion units if any)? I think all of them are occupied, There are free slots but I think they are fillers only with no PCI slots inside. We have 5 cards, the two Ethernet adapter, two 2 port FC adapters directly attached to a FS5200 and last one is a SAS Adapter to plug the LTO9 tape drive inside a 7226 rackmount.

And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Yes, that's correct. I have port number 2 in card 1 as a SEA and have port number 2 in card 2 also, so the LPAR has two ETHLINEs defined, one to the SEA of card one and the other, to the SEA of card two.

The issue is that the switch connected to card one was powered off, and the VIRTUALIP didn't be aware of that, so the communications were interrupted.

So, my question is: Is there any posibility of configuration that don't need to reconfigure all VIO Server? SR-IOV can be configured as only one port (of total of 4 ports) or is only supported as a whole card? If SR-IOV would only be configured as a port basis, it could be great, because I could deconfigure SEA and use the port only to be shared to a LPAR. Also, the other 2 LPARs also could be connected via SR-IOV to each port per LPAR, having all LPAR dedicated and not shared ethernet ports in an ideal situation (because port 3 and 4 of echa ethernet card are unused right now).

Thank you for your help, I'm looking forward to waiting from your reply.

Best Regards,

Urtzi Larrieta

Yes, several LPARs can share an SR-IOV capable ethernet adapter without the need of a VIOS. The sharing functionality of such an adapter is similar to the one we had in the past with IVE (Integrated Virtual Adpaters) if you are more familiar with this term.

However, not all ethernet adapters are SR-IOV capable, and installing those which support it depends on the server type.

So, to make further progress on your need, more information is needed.

What is the type/model of the server? Maybe this is a Power10 S1022 9105-22A? Is this another one?
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)?
Do you have empty slots in the server (or expansion units if any)?
And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Maybe a quick schema would help understanding your setup.

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

In order to be more efficient, I did and attached a couple of drawing.

The first page is your current setup, as far as I understand it based on your last post. Don't hesitate to comment and post corrections.

The second page is about a possible solution, depending on the network switches you run. It is using ethernet link aggregations, with stacked switches. Stacked switches allow them to act a single logical switch, and therefore allowing aggregations to spread over physical switches. Using this setup, if you loose a switch (or a cable), the aggregation continues to work through the remaining switch (or cables). The requirement is to have switches capable to use this stacking feature. I used this in the past with Cisco switches, during 10 years or so. However we faced only one issue during this timeframe, when we had to upgrade Cisco OS on both switches at the same time, leading to a planned downtime. It was several years ago, so maybe this is no longer an issue. Note that aggregation provides redundancy and bandwidth increase as well. With this aggregation setup, you would have to:

• setup the network switches stacking link
• install two more cables on each ethernet adapter
• configure the aggregation on the switch
• configure the aggregation on the VIOS
• setup a new SEA using the aggregation as physical adapter
• setup a new virtual adapter on the VIOS to a virtual switch (there is no more a dual switch like you did)
• setup a new virtual adapter on each IBM i LPAR to the same virtual switch
• setup the current physical IP address (with the appropriate subnet) of each IBM i LPAR to the new virtual ethernet adapter ; there is no more virtual IP with this case

The third page is about installing SR-IOV capable ethernet adapters, as both #5260 and #5899 adapters do not support SR-IOV. The most up-to-date list of such adapters for the S1014 is in the sales manual (https://www.ibm.com/docs/en/announcements/power-s1014-9105-41b). There is also a valuable FAQ where you can find these adapters (https://community.ibm.com/community/user/power/viewdocument/sr-iov-vnic-and-hnv-information?CommunityKey=71e6bb8a-5b34-44da-be8b-277834a183b0&tab=librarydocuments), review the chapter "What Power10 based system I/O adapters support SR-IOV shared mode?". If you have two available slots in your server, you can investigate about which adapter could fit into your switches. SR-IOV provides a virtualization layer which allows sharing one physical port by several LPARs, without using VIOS. You create logical port (or Virtual Function) from the HMC and the server firmware creates a direct link from the adapter to the partition through a "physical-like" ethernet adapter. In that case you continue using VIPAs. With this setup, you would have to:

• install and cable two new SR-IOV adapters
• configure each one to be a shared adapter
• configure each Virtual Function for each IBM i LPAR
• on each IBM i LPAR, move each virtual IP to each newly created "physical-like" ethernet adapter

This is a first starting point of an infrastructure which prevents the issue you were talking about. Make sure to validate it in your context. I believe there are other possibilities (for instance, with a mix of one SR-IOV adapter and one virtual adapter through one SEA). Hopefully, other community members will help.

Hello:

Thank you for your fast response. I'll try to answer the questions you sent me:

What is the type/model of the server? Power10 S1014 model 9105-41B
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? We have 2 cards 1 Gigabit Ethernet (UTP) 4 Port Adapter PCIE-4x/Short model 5899, 5260, one in slot U78DB.ND0.WZS03NR-P0-C10 and the other one in slot U78DB.ND0.WZS03NR-P0-C9

How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)? Yes, we have 2 cards with 4 port each. The first port on the first adapter is the VIOS en0 adapter communication, the rest of ports are SEA (one SEA per phisical port and adapter).

Do you have empty slots in the server (or expansion units if any)? I think all of them are occupied, There are free slots but I think they are fillers only with no PCI slots inside. We have 5 cards, the two Ethernet adapter, two 2 port FC adapters directly attached to a FS5200 and last one is a SAS Adapter to plug the LTO9 tape drive inside a 7226 rackmount.

And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Yes, that's correct. I have port number 2 in card 1 as a SEA and have port number 2 in card 2 also, so the LPAR has two ETHLINEs defined, one to the SEA of card one and the other, to the SEA of card two.

The issue is that the switch connected to card one was powered off, and the VIRTUALIP didn't be aware of that, so the communications were interrupted.

So, my question is: Is there any posibility of configuration that don't need to reconfigure all VIO Server? SR-IOV can be configured as only one port (of total of 4 ports) or is only supported as a whole card? If SR-IOV would only be configured as a port basis, it could be great, because I could deconfigure SEA and use the port only to be shared to a LPAR. Also, the other 2 LPARs also could be connected via SR-IOV to each port per LPAR, having all LPAR dedicated and not shared ethernet ports in an ideal situation (because port 3 and 4 of echa ethernet card are unused right now).

Thank you for your help, I'm looking forward to waiting from your reply.

Best Regards,

Urtzi Larrieta

Yes, several LPARs can share an SR-IOV capable ethernet adapter without the need of a VIOS. The sharing functionality of such an adapter is similar to the one we had in the past with IVE (Integrated Virtual Adpaters) if you are more familiar with this term.

However, not all ethernet adapters are SR-IOV capable, and installing those which support it depends on the server type.

So, to make further progress on your need, more information is needed.

What is the type/model of the server? Maybe this is a Power10 S1022 9105-22A? Is this another one?
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)?
Do you have empty slots in the server (or expansion units if any)?
And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Maybe a quick schema would help understanding your setup.

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Hello:

What a work you've done! The drawio examples in the current situation are exactly the current infraestructure now. I must to ask the customer's network guys if those switches connected to the POWER10 are LACP capable, because the switches in the rack I think they are plain interconnect switches only and maybe can not be configured in that way, but I must be sure if is the case or not.

The last resort is buying a new Ethernet interface card with SR-IOV capable, that's correct?

Thank you for all work you've done. One final question: Can you send me a guide where I can read how to configure LACP inside VIOS with the environment you suggested in the Drawio images?

In order to be more efficient, I did and attached a couple of drawing.

The first page is your current setup, as far as I understand it based on your last post. Don't hesitate to comment and post corrections.

The second page is about a possible solution, depending on the network switches you run. It is using ethernet link aggregations, with stacked switches. Stacked switches allow them to act a single logical switch, and therefore allowing aggregations to spread over physical switches. Using this setup, if you loose a switch (or a cable), the aggregation continues to work through the remaining switch (or cables). The requirement is to have switches capable to use this stacking feature. I used this in the past with Cisco switches, during 10 years or so. However we faced only one issue during this timeframe, when we had to upgrade Cisco OS on both switches at the same time, leading to a planned downtime. It was several years ago, so maybe this is no longer an issue. Note that aggregation provides redundancy and bandwidth increase as well. With this aggregation setup, you would have to:

• setup the network switches stacking link
• install two more cables on each ethernet adapter
• configure the aggregation on the switch
• configure the aggregation on the VIOS
• setup a new SEA using the aggregation as physical adapter
• setup a new virtual adapter on the VIOS to a virtual switch (there is no more a dual switch like you did)
• setup a new virtual adapter on each IBM i LPAR to the same virtual switch
• setup the current physical IP address (with the appropriate subnet) of each IBM i LPAR to the new virtual ethernet adapter ; there is no more virtual IP with this case

The third page is about installing SR-IOV capable ethernet adapters, as both #5260 and #5899 adapters do not support SR-IOV. The most up-to-date list of such adapters for the S1014 is in the sales manual (https://www.ibm.com/docs/en/announcements/power-s1014-9105-41b). There is also a valuable FAQ where you can find these adapters (https://community.ibm.com/community/user/power/viewdocument/sr-iov-vnic-and-hnv-information?CommunityKey=71e6bb8a-5b34-44da-be8b-277834a183b0&tab=librarydocuments), review the chapter "What Power10 based system I/O adapters support SR-IOV shared mode?". If you have two available slots in your server, you can investigate about which adapter could fit into your switches. SR-IOV provides a virtualization layer which allows sharing one physical port by several LPARs, without using VIOS. You create logical port (or Virtual Function) from the HMC and the server firmware creates a direct link from the adapter to the partition through a "physical-like" ethernet adapter. In that case you continue using VIPAs. With this setup, you would have to:

• install and cable two new SR-IOV adapters
• configure each one to be a shared adapter
• configure each Virtual Function for each IBM i LPAR
• on each IBM i LPAR, move each virtual IP to each newly created "physical-like" ethernet adapter

This is a first starting point of an infrastructure which prevents the issue you were talking about. Make sure to validate it in your context. I believe there are other possibilities (for instance, with a mix of one SR-IOV adapter and one virtual adapter through one SEA). Hopefully, other community members will help.

Hello:

Thank you for your fast response. I'll try to answer the questions you sent me:

What is the type/model of the server? Power10 S1014 model 9105-41B
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? We have 2 cards 1 Gigabit Ethernet (UTP) 4 Port Adapter PCIE-4x/Short model 5899, 5260, one in slot U78DB.ND0.WZS03NR-P0-C10 and the other one in slot U78DB.ND0.WZS03NR-P0-C9

How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)? Yes, we have 2 cards with 4 port each. The first port on the first adapter is the VIOS en0 adapter communication, the rest of ports are SEA (one SEA per phisical port and adapter).

Do you have empty slots in the server (or expansion units if any)? I think all of them are occupied, There are free slots but I think they are fillers only with no PCI slots inside. We have 5 cards, the two Ethernet adapter, two 2 port FC adapters directly attached to a FS5200 and last one is a SAS Adapter to plug the LTO9 tape drive inside a 7226 rackmount.

And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Yes, that's correct. I have port number 2 in card 1 as a SEA and have port number 2 in card 2 also, so the LPAR has two ETHLINEs defined, one to the SEA of card one and the other, to the SEA of card two.

The issue is that the switch connected to card one was powered off, and the VIRTUALIP didn't be aware of that, so the communications were interrupted.

So, my question is: Is there any posibility of configuration that don't need to reconfigure all VIO Server? SR-IOV can be configured as only one port (of total of 4 ports) or is only supported as a whole card? If SR-IOV would only be configured as a port basis, it could be great, because I could deconfigure SEA and use the port only to be shared to a LPAR. Also, the other 2 LPARs also could be connected via SR-IOV to each port per LPAR, having all LPAR dedicated and not shared ethernet ports in an ideal situation (because port 3 and 4 of echa ethernet card are unused right now).

Thank you for your help, I'm looking forward to waiting from your reply.

Best Regards,

Urtzi Larrieta

Yes, several LPARs can share an SR-IOV capable ethernet adapter without the need of a VIOS. The sharing functionality of such an adapter is similar to the one we had in the past with IVE (Integrated Virtual Adpaters) if you are more familiar with this term.

However, not all ethernet adapters are SR-IOV capable, and installing those which support it depends on the server type.

So, to make further progress on your need, more information is needed.

What is the type/model of the server? Maybe this is a Power10 S1022 9105-22A? Is this another one?
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)?
Do you have empty slots in the server (or expansion units if any)?
And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Maybe a quick schema would help understanding your setup.

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------

Hello

Configuring lacp is quite simple. It might be possible and even simpler from the HMC GUI, but I cannot confirm this.

• take a note of entx devices names that will be members of the aggregation; in the setup I suggest you will have 4 entx devices
• create the aggregation device with mkvdev command (https://www.ibm.com/docs/en/power10/9105-41B?topic=dc-mkvdev-command) and specify the aggregation mode (most often 802.3ad will apply) which must match the one you have on the switches
  • mkvdev -lnagg entxx entxy entyx entyy mode=8023ad
  • this will create a new entx device
  • chkout the aggregation status with entstat command (https://www.ibm.com/docs/en/power10/9105-41B?topic=nc-entstat-command) against the newly created device; I am sorry but I do not remember what are the expected status
• use the newly created device as the physical adapter of the SEA, just like if it was a single adapter port device

Regarding the SR-IOV adapters, here is the list of supported features (coming from the FAQ I was talking earlier) on Power10 models:

So, your choice is between #EC2U/EC2T, #EC66/EC67, #EC76/EC75, #EC72/EC71 and #EN26/EN24, depending on the firmware level of the server.

Now, checkout the hardware documentation (entry point here https://www.ibm.com/docs/en/power10/9105-41B?topic=adapters-adapter-details) of each adapter to review the switch and cables requirement.

Example for #EC2U/EC2T: https://www.ibm.com/docs/en/power10/9105-41B?topic=ad-pcie3-2-port-2510-gb-nic-roce-sfp28-adapter-fc-ec2t-fc-ec2u-ccin-58fb

Review carefully all the details and attributes, as for instance for the EC2U/EC2T you can read:

----- quote ------

Attributes provided

• RDMA over Converged Ethernet (RoCE).
• Dual port 25 Gb/10 Gb Ethernet network connectivity.
• Supports 25 Gb Ethernet SFP28 connectivity.Supports 10 Gb Ethernet SFP+ connectivity.
• Supports 25 Gb SFP28 SR connectivity with a 25 Gb optical transceiver (IBM® P/N 03GH278, purchased separately).
• Supports 10 Gb SFP+ SR connectivity with a 10 Gb optical transceiver (IBM P/N 01FT829, purchased separately).
• Supports 1 Gb RJ45 connectivity with a 1000BASE-T transceiver (IBM P/N 03FP283, purchased separately). Supports 1 Gb and full-duplex only. PowerVM® SR-IOV not supported for 1 Gb.
• AIX® Network Installation Management (NIM) support.
• PCI Express 3.0 (up to 8 GT/s) x8.PCIe Gen 3.0 compliant, 1.1 and 2.0 compatible.IEEE 802.3ae (25 Gb or 10 Gb Ethernet), IEEE 802.3ad (Link Aggregation & Failover), IEEE 802.3az (Energy Efficient Ethernet), IEEE 802.1Q/P (VLAN Tagging), IEEE 802.10au (Congestion Notification), IEEE 802.1Qbg, IEEE 802.3Qaz D0.2 (ETS), IEEE 802.1Qbb D1.0 (PFC), IEEE 1588v2 (PTP).
• Jumbo frame support up to 9.6 KB.
• VXLAN and NVGRE Overlay Network offload support.
• TCP/UDP/IP stateless offload.
• TCP/UDP checksum offload.
• TCP segmentation offload.
• PowerVM SR-IOV support for 10 Gb and higher speeds.

----- quote ------

Finally it might be possible to setup redundancy with some Hybrid Network Virtualization environments (maybe with one SR-IOV adapter and one SEA through a VIOS), but I am not able to provide more details. You may want to review https://community.ibm.com/community/user/power/blogs/charles-graham1/2020/06/19/hybrid-network-virtualization-using-sr-iov-for-opt.

For your convenience, I attach the Draw.io document.

Hello:

What a work you've done! The drawio examples in the current situation are exactly the current infraestructure now. I must to ask the customer's network guys if those switches connected to the POWER10 are LACP capable, because the switches in the rack I think they are plain interconnect switches only and maybe can not be configured in that way, but I must be sure if is the case or not.

The last resort is buying a new Ethernet interface card with SR-IOV capable, that's correct?

Thank you for all work you've done. One final question: Can you send me a guide where I can read how to configure LACP inside VIOS with the environment you suggested in the Drawio images?

In order to be more efficient, I did and attached a couple of drawing.

The first page is your current setup, as far as I understand it based on your last post. Don't hesitate to comment and post corrections.

The second page is about a possible solution, depending on the network switches you run. It is using ethernet link aggregations, with stacked switches. Stacked switches allow them to act a single logical switch, and therefore allowing aggregations to spread over physical switches. Using this setup, if you loose a switch (or a cable), the aggregation continues to work through the remaining switch (or cables). The requirement is to have switches capable to use this stacking feature. I used this in the past with Cisco switches, during 10 years or so. However we faced only one issue during this timeframe, when we had to upgrade Cisco OS on both switches at the same time, leading to a planned downtime. It was several years ago, so maybe this is no longer an issue. Note that aggregation provides redundancy and bandwidth increase as well. With this aggregation setup, you would have to:

• setup the network switches stacking link
• install two more cables on each ethernet adapter
• configure the aggregation on the switch
• configure the aggregation on the VIOS
• setup a new SEA using the aggregation as physical adapter
• setup a new virtual adapter on the VIOS to a virtual switch (there is no more a dual switch like you did)
• setup a new virtual adapter on each IBM i LPAR to the same virtual switch
• setup the current physical IP address (with the appropriate subnet) of each IBM i LPAR to the new virtual ethernet adapter ; there is no more virtual IP with this case

The third page is about installing SR-IOV capable ethernet adapters, as both #5260 and #5899 adapters do not support SR-IOV. The most up-to-date list of such adapters for the S1014 is in the sales manual (https://www.ibm.com/docs/en/announcements/power-s1014-9105-41b). There is also a valuable FAQ where you can find these adapters (https://community.ibm.com/community/user/power/viewdocument/sr-iov-vnic-and-hnv-information?CommunityKey=71e6bb8a-5b34-44da-be8b-277834a183b0&tab=librarydocuments), review the chapter "What Power10 based system I/O adapters support SR-IOV shared mode?". If you have two available slots in your server, you can investigate about which adapter could fit into your switches. SR-IOV provides a virtualization layer which allows sharing one physical port by several LPARs, without using VIOS. You create logical port (or Virtual Function) from the HMC and the server firmware creates a direct link from the adapter to the partition through a "physical-like" ethernet adapter. In that case you continue using VIPAs. With this setup, you would have to:

• install and cable two new SR-IOV adapters
• configure each one to be a shared adapter
• configure each Virtual Function for each IBM i LPAR
• on each IBM i LPAR, move each virtual IP to each newly created "physical-like" ethernet adapter

This is a first starting point of an infrastructure which prevents the issue you were talking about. Make sure to validate it in your context. I believe there are other possibilities (for instance, with a mix of one SR-IOV adapter and one virtual adapter through one SEA). Hopefully, other community members will help.

Hello:

Thank you for your fast response. I'll try to answer the questions you sent me:

What is the type/model of the server? Power10 S1014 model 9105-41B
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? We have 2 cards 1 Gigabit Ethernet (UTP) 4 Port Adapter PCIE-4x/Short model 5899, 5260, one in slot U78DB.ND0.WZS03NR-P0-C10 and the other one in slot U78DB.ND0.WZS03NR-P0-C9

How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)? Yes, we have 2 cards with 4 port each. The first port on the first adapter is the VIOS en0 adapter communication, the rest of ports are SEA (one SEA per phisical port and adapter).

Do you have empty slots in the server (or expansion units if any)? I think all of them are occupied, There are free slots but I think they are fillers only with no PCI slots inside. We have 5 cards, the two Ethernet adapter, two 2 port FC adapters directly attached to a FS5200 and last one is a SAS Adapter to plug the LTO9 tape drive inside a 7226 rackmount.

And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Yes, that's correct. I have port number 2 in card 1 as a SEA and have port number 2 in card 2 also, so the LPAR has two ETHLINEs defined, one to the SEA of card one and the other, to the SEA of card two.

The issue is that the switch connected to card one was powered off, and the VIRTUALIP didn't be aware of that, so the communications were interrupted.

So, my question is: Is there any posibility of configuration that don't need to reconfigure all VIO Server? SR-IOV can be configured as only one port (of total of 4 ports) or is only supported as a whole card? If SR-IOV would only be configured as a port basis, it could be great, because I could deconfigure SEA and use the port only to be shared to a LPAR. Also, the other 2 LPARs also could be connected via SR-IOV to each port per LPAR, having all LPAR dedicated and not shared ethernet ports in an ideal situation (because port 3 and 4 of echa ethernet card are unused right now).

Thank you for your help, I'm looking forward to waiting from your reply.

Best Regards,

Urtzi Larrieta

Yes, several LPARs can share an SR-IOV capable ethernet adapter without the need of a VIOS. The sharing functionality of such an adapter is similar to the one we had in the past with IVE (Integrated Virtual Adpaters) if you are more familiar with this term.

However, not all ethernet adapters are SR-IOV capable, and installing those which support it depends on the server type.

So, to make further progress on your need, more information is needed.

What is the type/model of the server? Maybe this is a Power10 S1022 9105-22A? Is this another one?
What are the feature codes of the ethernet adapters? Something like #5260, #5899, #ENxx? How many ethernet adapters (talking about adapters not ports) do you have (at least 2, I guess)?
Do you have empty slots in the server (or expansion units if any)?
And about the actual setup, my understanding is that you have two SEA in your VIOS using two distinct ethernet ports, connected to two distinct network switches. Is that correct? Maybe a quick schema would help understanding your setup.

Hello again, I didn't see this message.

SRV-IO can share the adapter between 3 LPARs? Now I have a SEA that shares the physical Ethernet adapter through the 3 LPARs, if SRV-IO can, it could be a solution.

Another solution, depending on your hardware, could be to use SR-IOV ethernet adapters. This way, you can keep on using your VIRTUALIP setup as those adapters are, indeed, physical ones and no longer using the VIOS and virtual switches.

Checkout https://www.ibm.com/docs/en/power10?topic=networking-single-root-io-virtualization

------------------------------
Marc Rauzier

Original Message:
Sent: Mon April 14, 2025 03:39 AM
From: Urtzi Larrieta Alvarez
Subject: IBMi *VIRTUALIP not working under VIOS

Hello:

Sorry if this question was been answered in another forum. We have a POWER10 with VIOS 3.1 and several IBMi LPAR working, all of them with two ETHLINE network interfaces attached to different Virtual Switches with SEA that are mapped to different physical 4-Port GbE network cards. Each LPAR has *VIRTUALIP configured between them, so if anything goes wrong in one of interface, the remaining interface would assume the production networking transparently.

That being said, we had a network/power outage issue, consisting of one of the two lines that power the rack where everything is working, and one of the two network switches went powered off.

In a POWER10 with IBMi standalone configuration, if one network interface become offline or unavailable, because the switch associated to it get unavailable, the *VIRTUALIP does the job, switching the round-robin traffic and routing all the traffic to the remaining physical interface and no one in the network get's rid of it. Once the issue is resolved, and the physical interface become ready again, the *VIRTUALIP starts to share the traffic between the two physical interfaces again automatically.

But in the VIOS with SEA, we had a power outage in one of the lines, thus one of the network switches became inoperative, but for the IBMi's *VIRTUALIP interface, did not detect that one of the physical interface went down, so *VIRTUALIP remained sharing traffic between two interfaces, resulting in many sessions get lost and having a production outage because the IBMi LPAR did not know that one of the physical interfaces became offline.

Inside VIOS, with errpt -a command, the ETHERNET DOWN record was registered, but I think that the Virtual Switch did not pass the new status of the interface (offline) to the LPAR, so the IBMi and his *VIRTUALIP did not know that one physical interface was down, so the *VIRTUALIP thought that the two interfaces were running when the reality was other.

That being said (sorry for my poor English), is there any configuration or method to pass the offline behaviour to the virtual interface attached to LPARs so the *VIRTUALIP can switch the traffic to the remaining interface and not interrupting traffic?

Thank you in advance. 

Best Regards,

Urtzi Larrieta

------------------------------
Urtzi Larrieta Alvarez
IT Manager
ECNA Informatica
Bilbao
944159688
------------------------------