Today almost none of this is new news, however, I'm reposting here to help ensure the information is more easily found. Hopefully I'll have some P11 to share soon.

------------------------------------------------------------------------------------------------------------------

Following are some lessons learned during a recent S1024 system install (Feb 2023).

Virtual Management IP (VMI)

This is new in HMC V10R1M1020. Since this is the only system our new 7063-CR2 HMC will manage we ran a direct cable connection straight from our designated private port on the HMC to the top/first eBMC port on the back of the S1024 as shown in pic below.  Though not completely relevant if it is direct or switch attached or not it’s just more of an FYI of our environment.

Upon connecting, the HMC DHCP server successfully assigned an IP address. The output shown below is after initial connection. Notice, which actually took me a minute to get my attention, that the numbers on the end are the DHCP IP address it was assigned.

We entered the HMC password to get it to authenticate and showed the system in standby node as normal. However, shortly thereafter it showed “No Connection”.  Wait, what? Why? When the mouse went over the message the pop-up of “Virtual management interface IP is not configured” is shown. Of course it is, right? Wrong!

So what is the VMI? Well that’s a great question. My personal goto, the redbooks, on P10 Scale Out systems only mentions this:

Well that’s only partially informative of what it is, and doesn’t show how to configure it.  Upon further searching with the HMC level of 1020 I found Hari’s blog explaining the new features  and one of them is the VMI.

His blog entry is here:

https://community.ibm.com/community/user/power/blogs/hariganesh-muralidharan1/2022/07/27/whats-new-in-hmc-10110200

It’s quite good at explaining what it is, yet still doesn’t tell where/how to set it.

Ok, time to search the HMC itself. When utilizing the search in M1020 nothing comes back. When doing so in M1030 the following info, in pic below, is shown. Which again tells about the options but not WHERE to set it.

So, I just break down and start looking around manually and FINALLY found it.  Now the GUI  does differ a bit between M1020 and M1030 and I provide pics of both.  The short of it is, it is located under System actions.

In M1020 select the system and then in top left, System Actions->Operations->VMI Configuration as shown in pic below left:

In M1030, pic above right, select the system and then in top right expand the “Systems actions” tab and scroll down “Connection and Operations” and click on “VMI connection”.

Once selected you are then displayed with the following screen that will show for both ports.

We choose the eth0 port, action, and set it as type “dynamic”. Though it does default to static as shown below:

Once completed it ultimately resolved our “No Connection” problem.

After originally posting this article, Andrey Klyachkin shared via Twitter the following link to fantastic EBMC videos.

https://mediacenter.ibm.com/channel/POWER10%2B%2BEBMC%2BVideo%2BSeries/257624232

The one on the top 3 things, includes the VMI and it is here.

Hypervisor Overhead

This brand new 9105-42A system with 24 cores/1TB memory/EMX0 expansion unit has consumed over 14.5GB in overhead before configuring a single partition. That seems like a lot from the start. The system also consists of following adapters:

• (4) EC2U 25/10 Gb NIC & RoCE SFP28 Adapters (SR-IOV capable)
• (4) EN1A 32Gb 2-port Fiber Channel PCIe3 Adapters
• (1) EN1C 16Gb 4-port Fiber Channel PCIe3 Adapter
• (1) EJ10 SAS 6Gb 4-Port PCIe3 Adapter
• (1) EJ2A Expansion I/O drawer Adapter

Upon further review, testing, and feedback from support line we have surmised the following.  Support previously informed us via another customer of “the two SRIOV adapters in shared mode use 5.25g”.  Enabling the EC2U adapters from dedicated to shared mode had NO effect on the overhead. There appears no way to get that overhead back. This at a rate of ~ 2.6GB per card.  Since we know there is about 5.25GB for two cards, and this environment has four of those cards, that explains ~10.5GB of the overhead from the very beginning. There is also another 1.25GB required for the VMI “hidden partition”. The DMA for the card slots is another ~2GB (not including expansion drawer, if applicable). That explains almost all of the reserved overhead.  Like or it not, that does work out about right.

So how would’ve we known this? Use the System Planning Tool (SPT). We did that. It’s all over the map. It says with our hardware config and with ONLY the two VIOS configured it would use 87.5GB. WHAT?! No way. Oh wait, it defaults to firmware memory mirror enabled, turned that off and now shows 42.5GB without any other LPARs built out. Once filling in all the LPAR sizes it comes back with ~63GB.

For comparison, in our real environment we configured all 21 lpars, with NPIV, VNIC, some VSCSI and used 43GB as shown in pic below.

That is a delta of almost 50%. But I suspect it’s better to estimate an overage than a shortage.  But still not particularly close.

1030.01(030) System Firmware

This new S1024 came preinstalled with fw1020.10 (85) as shown below.

A newer version, 1030.01(030) is already available so of course we want to implement the latest and greatest.

Following below is the special instructions that come with the updated firmware level. It is important and yet only partially helpful. I’ll explain why as I learned, well relearned I suppose is more correct, the hard way.

First off it says, “Concurrent Service Pack”. This normally implies it is non-disruptive, that is absolutely not the case. As shown in the following preview screen before installing it clearly says it’s disruptive.

Secondly because of a known issue it says you must install it twice consecutively, which is completely correct. As the level shown after the first install follows below you can see the install and activated levels differ.

What is not explicitly called out in the special instructions that the HMC version must be a minimum of V10R2M1030.  The “Check Readiness” prior to performing a firmware update has nothing to do with checking the target level at all, it’s doesn’t even know what you are going to update to just if the system is in a state of Ready to perform an update. You can still install the firmware update the first time, to complete it and then lose access to the managed system with the dreaded “Version Mismatch” message as shown below:

Though we already had plans to updating HMC to latest and greatest we weren’t planning on doing so right this minute, but that has changed. Note, it’s not an update, it’s an upgrade, so it too is completely disruptive. But upgrading to V10R2M1030 did indeed resolve the mismatch.

Upon further review, thanks Tsvetan Marinov for the reminder, the HMC M1030 requirement is clearly covered in the 01ML1030_026_026.html  file. But for reasons I can’t explain why we didn’t get that file in our download as shown below.

Contents of that html file can also be found here. The main bit about the HMC version required is seen below:

A long history of experience means I should’ve known better. But those new to it, may not know at all.

Managed System(s) in “Recovery” status

While installing 15 S1022s, (Firmware 1050.x) a few months back, three of them came up in the “Recovery” state to the HMC. I personally had never seen or experienced this. The common denominator on these three is they were initially powered on manually (via the power button), not via the HMC. Unsure how or why exactly that matters , and since then I believe I saw a firmware, or maybe it was HMC, bug description that sounded similar to this issue maybe it won’t be an issue any longer.

Initially I thought it was a bad result of connecting to HMC not high enough to support the system. But I had 10 initially connected to HMC 1030, and the other 5 were connected to HMC 1050. However, we had two of the 10 error, and one of the 5 so that shouldn’t have been it. Again the only known common denominator was they were powered up manually, regardless of HMC level.

But then when HMC level was rectified, powering on the the other 12 came on just fine. Those 12 never had an error before during or after. Which is strange because none of the 15 ever really established a valid connection, yet were seen by the HMCs. Just the ones that powered on initially got corrupted some how.  I would think corrupting the firmware is a bit harsh for the situation. But as much as I think I’m special on corrupting stuff, I’ve come to realize this problem isn’t unique to me.

For the three in the “Recovery” state I found this procedure :

https://www.ibm.com/docs/en/power9?topic=hmc-recovering-partition-data-managed-system

Yet in this case it doesn’t apply/can’t be used because they are new, never been seen by HMC so no backup data on the HMC to recover from. So the only other method is to reset system settings via BMC port. This does wipe out all partition data, but other than the pesky service partition we have nothing to lose anyway. The procedure is listed at the end of this section. But I also want to share other problem instances that this procedure has become the magic cure all.

I since have remembered reading a situation from Jaqui Lynch where a system came in some default factory build mode and it too had to be reset. Leading to a similar “recovery”  set of steps.

Connection Failed, Firmware corrupted

In late August 2024, we had 2 of 3 L1022s that also had to be factory reset. After initial connection, power on, password, did the VMI, then later the eBMC (ASMI) showed no connection (or connection failedl I don’t really remember which one). At some point we encountered a, Firmware was corrupted, message and needed to be reloaded. This same reset did appear to fix it.

New E1050 Power On failure

Late September, new install with HMC v1060 and system firmware level 1060.x we had one of eight E1050s that wouldn’t power on. After about ~3minutes it would hit just display if failed to power on. It powers up, ramps up the fans, for ~40 seconds, fans spin down quickly and shuts down to standb. It then waits ~20 seconds, then it tries again. This occurs three times. The last time it just turns on the LED, keeps the fans up high, and leaves it in the error state.

I don’t know a good reason to leave it running that way so I told it to power off. I unplugged for 10 minutes, tried again, same result. I unplugged, popped open the lid, with gloves lightly pressed down and around to see if I could find anything obviously loose, and nothing. Though I could hear and see all four fans working, and they were installed onsite, I thought man is one of them not making a good connection. But nope, that’s not it. After engaging support, it too appears this same reset procedure fixed it. I can’t say I expected that to be the resolution.

The reset procedure was:

1. Login to the ASMI
2. Power Off server
3. Go into settings, reset server settings
4. There are two options:
a. Reset server settings: Resets the server settings only.
b. Reset BMC and server settings: Resets both the server and the BMC settings.
5. Select – a. Reset server settings only
6. Under power settings make sure it is set to user initiated (standby)
7. Check LMB is set to 256MB
8. Log out of ASMI
9. Go to the HMC GUI
10. Reconfigure the VMI

MISC Factory Inconsistencies

Adapter placement

On occasion we are getting I/O cards not placed in the adapter placement recommended best practices order. I’m not saying I’m moving cards on every install, but it is greater than zero. Though non-SR-IOV slots are more and more rare, we’ve had to move some for that reason . Seems to be primarily for IBM I we had to move to SR-IOV slots. Some of it isn’t even swapping cards/slots, some is just a pure move to both a preferred and SR-IOV capable one.

Mismatch adapter microcode

I’ve also had like adapters come with different levels of microcode. Overall that one is rare, and could simply just be some from different batches. I’ve only hit that once in the last year or so. But would be nice if they were uniform/checked/the same from the factory. Maybe make real use of that otherwise service partition.

Quality Control

I have encountered some odd QC stuff. For instance the R logo stamped on front latch of the E1050 upside down. One of three was this way. There is only one way they go on so not even I can put it on upside down. But just looks very lackluster. So don’t stare at it after it’s installed.

Which HMC port comes factory covered with a tab varies. This is definitely not a technical problem but still just an odd example of differences. I only noticed because I was having a connection problem to one. I pulled the IP off the panel, hard wired in and couldn’t communicate with it. For some, embarrassingly enough, 10 minutes later notice I wasn’t plugged into the HMC1 port at all it was still covered. But on the other two systems the HMC2 port was covered. I actually do run into this one quite a bit. Because once it happened I made a mental note to always look now. For dual HMCs I take the tab off anyway, so it doesn’t matter, but that’s less maybe 20% of the time I have dual HMCs. So If I tell someone else to cable the free/open/uncovered port, they can be unintentionally mismatched.

Missing all four of the main tiny screws from the HMC kit for rail attachment to the HMC. They are unique and not plentiful to find more from other device kits. We weren’t missing one, but all. In my case we had dual HMCs so I took half the screws from the other HMC kit and it was fine.

Historically problems with the kits, factory configurations, hardware are extremely few and far between. I believe in the last year I’ve hit more than usual. Nothing awful, show stopping issues. But it does delay things from progressing.