HMC

 View Only
Expand all | Collapse all

Remote HMC disconnect during a managed system firmware update

  • 1.  Remote HMC disconnect during a managed system firmware update

    Posted 19 days ago

    Hi folks, 

    I am seeking confirmation on the implications of a remote HMC connection (over a VPN connection) failing while a managed system firmware update initiated by this remote HMC is in progress. 

    A question submitted to ChatGPT provided the following response:

    In summary, while losing the remote connection to the HMC during a firmware update or upgrade can limit your ability to monitor and control the process, the update or upgrade itself should continue as long as the local HMC console is active. The local HMC console does not need to be signed on for the update or upgrade to proceed, but having it signed on can be helpful for monitoring and troubleshooting. Ensure you have contingency plans and clear communication channels with local staff to handle any potential issues.

    Is anyone able to verify please the validity of that response?

    Has anyone experienced this situation? 

    Thank you,

     



    ------------------------------
    ========================
    Jozsef Torok
    IT Engineer - IBM i and Power
    Spark New Zealand Trading Ltd
    ========================
    ------------------------------


  • 2.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 19 days ago

    I expect that once the firmware is transferred to the FSP, all the remaining steps (verification/update) are internal and managed by the FSP and PHYP, but if you need it in writing, you'll need to open a case.

    Note that HMC<->FSP over a VPN isn't exactly a supported scenario.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 3.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 17 days ago

    Hi José, thank you for your response. 
    It makes sense that once the transfer of the update/upgrade files has completed (from a remote SFTP server) then at that stage the remote HMC 'only' provides a monitor of progress.
    From further research it looks to confirm that the update process is handled internally by the Power system's PHYP and FSP. The PHYP coordinates the update process and the FSP handles the storage and execution of the firmware update, including error detection and recovery mechanisms. 
    That to me means that if the FSP detects and reports an error it requires an active HMC to be able to report the error and allow for a response to the error. In such a situation where the remote HMC connection has been lost but the local HMC is active (not signed on), is it still possible to respond to the error once a remote HMC connection is restored, or else go onsite to sign on at the local HMC and respond to the error?  And how would the error be visible, as a serviceable event?

    When you say "HMC<->FSP over a VPN isn't exactly a supported scenario.", I should have stated earlier that we remote in to the company's network over a VPN connection. Once connected to the company IVDN we can start a remote HMC connection.
    Note that currently we only implement a firmware from our office when connected directly to our company's IVDN. This is to remove that risk of a VPN connection dropping when connecting from home.    

    So many questions.  I think it does warrant a support case. 

    Thank you,

    Jozsef



    ------------------------------
    =====================
    Jozsef Torok
    =====================
    ------------------------------



  • 4.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 19 days ago

    To remove this kind of risks, you may want to connect to a local HMC through IPMI for x86 based HMC or BMC for Power based HMC. Using this way, you initiate any process just like if you were close to the local HMC.



    ------------------------------
    Marc Rauzier
    ------------------------------



  • 5.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 17 days ago

    Hi Marc, thank you for responding. 

    I take it you are referring to Console Redirection via the HMC's IPMI?

    If we did that remotely via a VPN connection don't we still have the same risk of the VPN connection dropping during an update install?  Or does running the update process as if working on the local HMC negate that where when the VPN connection drops the process on the 'local' Console carries on?   

    Thanks Marc,

    Jozsef



    ------------------------------
    =====================
    Jozsef Torok
    IT Engineer
    Spark New Zealand Trading Ltd
    =====================
    ------------------------------



  • 6.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 16 days ago
    Edited by Marc Rauzier 16 days ago

    Hello Jozsef

    The availability of this BMC feature might depend on your HMC model, but, take a look at "Section 1. OpenBMC Consoles, Keyboard Video Mouse (KVM)" section on https://www.ibm.com/support/pages/openbmc-consoles-and-virtual-media-7063-cr2-hmc You will see that there is the HMC local login screen.

    You can get remote full control with the BMC feature, including "mirroring the local console", as they write. You connect to the BMC with a browser and initiate any activity (such as an HMC upgrade including reboots, for example) you want on the HMC just like if you were locally residing. If you loose the access to the BMC, or even if you disconnect, any action previously initiated will continue.

    This is a nice feature for IBM i users so that they can start a shared local 5250 console on a partition in order to remove the risk of loosing the connection to the 5250 console during vital operations such as IBM i upgrades, and remotely connect to this shared 5250 console to be more comfortable.



    ------------------------------
    Marc Rauzier
    ------------------------------



  • 7.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 16 days ago

    I am using vHMC or virtual HMC.  I have zero complaints about that.  Our data centers are geographically remote from me.  One is two hours south and the other is 3 hours north.

    I got tired of playing the upgrade game of trying to keep the physical HMC in step with the latest version of HMC.  And the hardware maintenance, etc.

    Now I use vHMC.

    • Prior to an HMC upgrade I can do a snapshot.  If it goes south I can restore the snapshot and do it right this time.
    • Using the web interface from my PC I can do most HMC functions.
    • For starting 5250 shared console sessions I can use the vmware server access to that box.  If you allocated 5GB of video memory (not more, not less) it works pretty well.
    • If I'm concerned about losing connection I use that same vmware connection.  My home internet is very unreliable.
    • I have redundant vmware servers at both DC's.  Each with a vHMC.  Either vHMC can access the Power server(s) in the data center it is in and the Power server(s) in the other DC.
    • With shared 5250 console sessions, and Console Takeover enabled, I can reboot the HMC and not experience any interruption in system console usage from my laptop.
    • ASMI remote is a breeze.

    Back when I have a physical box for HMC I could access that directly by plugging the keyboard, video and mouse into a KVM with remote access.  It was a little odd however with weird things like dual mouse pointers appearing on the screen.

    I simply would not go back to a physical HMC.

    There are those, however, who feel, while it is ok to have Personnel info, budget info, proprietary engineering designs, customer pricing negotations, missle launch codes, etc on the corporate network it is curtains for the free world if they don't have a dedicated wire between their HMC and their Power system.  I just can't get my head around that mentality.



    ------------------------------
    Robert Berendt IBMChampion
    Business Systems Analyst, Lead
    Dekko
    Fort Wayne
    ------------------------------



  • 8.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 14 days ago

    > I am using vHMC or virtual HMC.

    Out of curiosity: how do you deal with the cases when IBM tech support people have to do something that requires them to use HMC console? Separate PC connected to some segment of the company network with FW rules allowing only access to selected ports on vHMC IP?



    ------------------------------
    Lech Szychowski
    ------------------------------



  • 9.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 14 days ago

    This is a good question. When I was working, we had two data centers, around 10 km one from the other, and we had one HMC in each site, each one acting as a redundant HMC for the other site. The data centers and IBM devices were IBM owned and operated with tons of IBM hardware (Systems z, Power Systems, Storage...), so that there were dedicated IBM CEs.

    They had such a requirement to have a physical HMC as close as possible to the servers so that they could handle their repair and install actions.



    ------------------------------
    Marc Rauzier
    ------------------------------



  • 10.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 14 days ago

    If they are onsite they have a few choices.  I normally have to go to that site too in the few cases that this has happened.

    Normally the first choice is they can use my laptop.  I'm normally there solely to work on this issue.  I normally don't like the wifi at the data center as then I would have to vpn into the equipment.  So we have a spare wire hanging off the switch that any employee who goes to the DC can use to hook up their laptop while they are there.  I've already had to pass through four retinal scanners just to get into the DC.  Then I have to use a physical metal key to get into the cage surrounding our racks and get access to that wire.

    One time, while the IBM guy was on site and I was out of state I got in remotely and he talked me through clicking stuff on my laptop.

    Let me ask you this:  If someone at your company wants to pull up customer history do you only let them use a separate PC on a special segment of the company network with FW rules only allowing access to special ports on your lan connection to your Power system?  No?  Then why the abject paranoia regarding the HMC port to your Power system?



    ------------------------------
    Robert Berendt IBMChampion
    Business Systems Analyst, Lead
    Dekko
    Fort Wayne
    ------------------------------



  • 11.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 14 days ago

    There is another option for onsite.  The vm server has a KVM on it.  A technician can pull out the keyboard, hit the right combination to get to the vmserver and select the display option for the instance of the vHMC and use it from there.  I prefer the ergonomics of using my laptop.



    ------------------------------
    Robert Berendt IBMChampion
    Business Systems Analyst, Lead
    Dekko
    Fort Wayne
    ------------------------------



  • 12.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 19 days ago

    Hi Jozsef,

    I have (unfortunately) already experienced this kind of event in the past. Although I can confirm that the upgrade itself completed well. Enough contingency was built in to not render the system firmware in an inconsistent state...

    HOWEVER is there no possibility to have an HMC locally connected? I believe there is still the possibility to upgrade the firmware using the VIO server (or any other LPAR on the system)



    ------------------------------
    Kristof Van Elshocht
    Technical Sales
    Core ICT
    Brasschaat
    ------------------------------



  • 13.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 17 days ago

    Hi Kristof, thank you for the response.  

    Weird as this sounds your misfortune gives me some comfort. 

    If I understand your question "is there no possibility to have an HMC locally connected?" correctly, we do have local HMCs at our [lights out] sites in the same rack as the Power systems. And Marc's response about using the IPMI Console Redirection may have some merit. 

    Implementing a system firmware update via an LPAR is an interesting proposition. Do you have more information on that, is it an IBM Idea? 

    Thanks Kristof,

    Jozsef



    ------------------------------
    =====================
    Jozsef Torok
    IT Engineer
    Spark New Zealand Trading Ltd
    =====================
    ------------------------------



  • 14.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 16 days ago

    Jozsef,

    IBM will always state that the firmware of an HMC-managed system is to be done using the HMC... However, the firmware can be upgraded from the OS also... Just keep in mind to make sure the compatibility between HMC version and system FW is respected.

    I understood you will be filing an IBM support case, interested to hear their feedback also :-)



    ------------------------------
    Kristof Van Elshocht
    Technical Sales
    Core ICT
    Brasschaat
    ------------------------------



  • 15.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 15 days ago

    Jozsef,

    <snip>Implementing a system firmware update via an LPAR is an interesting proposition. Do you have more information on that, is it an IBM Idea? </snip>

    Quite easily, for those environments with IBM i.  However, even IBM i shops are told strongly that if you have an HMC you REALLY should not set your system up that way.  The reason being that you could upgrade your firmware past a level that your HMC can support.  It's really only recommended for shops without an HMC to make it simple so they do not have to use ASMI to upgrade their firmware.

    Read both of these:

    Matrix of HMC to firmware compatibility.  Find your model and click on View underneath Power Code Matrix:

    https://esupport.ibm.com/customercare/flrt/mtm#tab_p10

    There's an IBM i command, DSPFMWSTS.  Since I have an HMC mine is set to

    Firmware update policy . . . . . . . . . . . . . . :   *HMC

    I know someone without an HMC.  Theirs is set to

    Firmware update policy . . . . . . . . . . . . . . :   *OPSYS

    See also:

    Getting server firmware fixes through IBM i for a system not managed by a management console - IBM Documentation

    or

    Getting server firmware fixes through AIX or Linux without a management console - IBM Documentation

    They all say you HAVE to use HMC.  "Lead me not into temptation..."

    If you know IBM i, and are used to putting on OS PTF's, this is such an incredibly simple thing.  Absolutely zero extra steps.  However, I agree with IBM and you should not do it this way if you have an HMC.



    ------------------------------
    Robert Berendt IBMChampion
    Business Systems Analyst, Lead
    Dekko
    Fort Wayne
    ------------------------------



  • 16.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 15 days ago

    And be careful because, as far as I know, all firmware updates are disruptive if the server is not HMC managed.

    It was my understanding that when the IBM i partition is powered down then restarted to apply a firmware update, the entire server was also powered off then on. But if I understand correctly, this server restart can be scheduled, according to https://www.ibm.com/support/pages/node/6983552.

    ---- quote ----

    Command Line *OPSYS Upgrade:
                              
    • SNDPTFORD PTFID((MH0#### '57339**')) 
      • DSPPTF
        • MH0#### status will be "Save file only"
    • LODPTF LICPGM(57339**) DEV(*SERVICE) SELECT(MH0####)   
      • DSPPTF
        • MH0#### status will be "Not Applied"
    • APYPTF LICPGM(57339**) SELECT(MH0####) DELAYED(*YES)  
      • DSPPTF
        • MH0#### IPL action will be "YES"

    • A full system restart is required for firmware to be applied
      • Does not need to be immediate, may be scheduled

    ---- quote ----



    ------------------------------
    Marc Rauzier
    ------------------------------



  • 17.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 16 days ago
    Edited by Patrick Hügli 16 days ago

    Hello Jozsef

    I would recommend to use a terminal multiplexer like screen while installing system firmware. Best practice is to start the screen terminal on your preferred jumphost running in the local DC env. first and then connect to the local HMC and begin with the firmware update from there.
    With this setting you are secure for any outside network connection issues and you are able to reconnect to the screen session still running on the jumphost doing the firmware update.



    ------------------------------
    Patrick Hügli
    ------------------------------



  • 18.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 13 days ago
    Edited by Jozsef Torok 12 days ago

    Thank you very much everyone.  It is great to be able to bounce thoughts and ideas around like this.  

    I have since been able to make IPMI Console Redirection work, and although the frame rate leaves something to be desired it is quite usable.   I don't know if that is because it is a Java based KVM console, we have 7063-CR1 HMCs, or some bottle neck on our network.   But it does work well for mirroring the local console, so if my remote connection should drop it won't impact any active function we are running  at that time. Excellent!  

    Thanks everyone,

    Jozsef

    --------------------------------
    =====================
    Jozsef Torok
    IT Engineer
    Spark New Zealand Trading Ltd
    =====================
    ------------------------------



  • 19.  RE: Remote HMC disconnect during a managed system firmware update

    Posted 9 days ago

    Risks of using IPMI on IBM Power Systems and OpenPower Systems

    https://www.ibm.com/docs/en/power9/9040-MR9?topic=interfaces-risks-using-ipmi-power-systems-openpower-systems

    Just to throw out some FUD...



    ------------------------------
    Robert Berendt IBMChampion
    Business Systems Analyst, Lead
    Dekko
    Fort Wayne
    ------------------------------