PowerVC

 View Only
Expand all | Collapse all

PowerVC 2.0.3 / RHEL9 Guest VM

  • 1.  PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Tue January 17, 2023 03:33 AM
    Hi,

    Anyone having already RHEL9 Golden image created for PowerVC?

    I created my image VM ; VM itself works perfectly as such...But then when I create an Image out of that and try to deploy the image; the new VM always ends up in Emergency shell.

    Most probably some issue with Multipath and / or LVM conf..
    Tried already:
    - set filtering on /etc/lvm/lvm.conf; but that seems to be deprecated on RHEL9 
    - Increased SCSI INQUIRY timeout in kernel config via grubby
    - Increased dev_loss_tmo value in multipath conf

    >> no help
     


    Console log below:

    [  OK  ] Finished Create Static Device Nodes in /dev.

             Starting Rule-based Manage…for Device Events and Files...

    [   12.625258] synth uevent: /devices/vio: failed to send uevent

    [   12.625276] vio vio: uevent: failed to send synthetic uevent

    [   12.629013] synth uevent: /devices/vio/4000: failed to send uevent

    [   12.629023] vio 4000: uevent: failed to send synthetic uevent

    [   12.629073] synth uevent: /devices/vio/4001: failed to send uevent

    [   12.629079] vio 4001: uevent: failed to send synthetic uevent

    [   12.629112] synth uevent: /devices/vio/4002: failed to send uevent

    [   12.629118] vio 4002: uevent: failed to send synthetic uevent

    [   12.629148] synth uevent: /devices/vio/4004: failed to send uevent

    [   12.629154] vio 4004: uevent: failed to send synthetic uevent

    [  OK  ] Finished Coldplug All udev Devices.

             Starting Wait for udev To …plete Device Initialization...

    [  OK  ] Started Rule-based Manager for Device Events and Files.

    [  OK  ] Finished Monitoring of LVM… dmeventd or progress polling.

             Starting Load Kernel Module configfs...

             Starting Load Kernel Module fuse...

    [  OK  ] Finished Load Kernel Module configfs.

    [  OK  ] Finished Load Kernel Module fuse.

    [   12.928513] pseries_rng: Registering IBM pSeries RNG driver

    [  OK  ] Finished Wait for udev To Complete Device Initialization.

             Starting Device-Mapper Multipath Device Controller...

    [  OK  ] Started Device-Mapper Multipath Device Controller.

    [  OK  ] Reached target Preparation for Local File Systems.

             Mounting /boot...

    [   13.748344] XFS (dm-2): Mounting V5 Filesystem

    [   14.085278] XFS (dm-2): Ending clean mount

    [  OK  ] Mounted /boot.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvHome.

    [DEPEND] Dependency failed for /home.

    [DEPEND] Dependency failed for Local File Systems.

    [DEPEND] Dependency failed for Mark… need to relabel after reboot.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvVarTmp.

    [DEPEND] Dependency failed for /var/tmp.

    [ TIME ] Timed out waiting for device …/mapper/vgRoot-lvVarLogAudit.

    [DEPEND] Dependency failed for /var/log/audit.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvVar.

    [DEPEND] Dependency failed for /var.

    [DEPEND] Dependency failed for Reco… System Boot/Shutdown in UTMP.

    [DEPEND] Dependency failed for Record Runlevel Change in UTMP.

    [DEPEND] Dependency failed for /var/log.

    [DEPEND] Dependency failed for Flus…Journal to Persistent Storage.

    [DEPEND] Dependency failed for RPC Pipe File System.

    [DEPEND] Dependency failed for rpc_pipefs.target.

    [DEPEND] Dependency failed for RPC …ice for NFS client and server.

    [DEPEND] Dependency failed for Load/Save Random Seed.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvTmp.

    [DEPEND] Dependency failed for /tmp.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvSrv.

    [DEPEND] Dependency failed for /srv.

    [ TIME ] Timed out waiting for device /dev/mapper/vgRoot-lvOpt.

    [DEPEND] Dependency failed for /opt.

    [  OK  ] Stopped Forward Password R…uests to Wall Directory Watch.

    [  OK  ] Reached target Timer Units.

    [  OK  ] Reached target NFS client services.

    [  OK  ] Reached target Login Prompts.

    [  OK  ] Reached target Cloud-init target.

    [  OK  ] Reached target Cloud-config availability.

    [  OK  ] Reached target Preparation for Network.

    [  OK  ] Reached target Network.

    [  OK  ] Reached target Network is Online.

    [  OK  ] Reached target Preparation for Remote File Systems.

    [  OK  ] Reached target Remote File Systems.

             Starting Notify NFS peers of a restart...

    [  OK  ] Reached target User and Group Name Lookups.

    [  OK  ] Reached target Socket Units.

    [  OK  ] Started Emergency Shell.

    [  OK  ] Reached target Emergency Mode.

             Starting Tell Plymouth To Write Out Runtime Data...

             Starting Set Up Additional Binary Formats...

             Starting Create Volatile Files and Directories...

             Mounting Arbitrary Executable File Formats File System...

    [  OK  ] Started Notify NFS peers of a restart.

    [  OK  ] Finished Tell Plymouth To Write Out Runtime Data.

    [  OK  ] Mounted Arbitrary Executable File Formats File System.

    [  OK  ] Finished Set Up Additional Binary Formats.

    [  OK  ] Finished Create Volatile Files and Directories.

             Starting Security Auditing Service...

             Starting RPC Bind...

             Starting Rebuild Journal Catalog...

    [FAILED] Failed to start Security Auditing Service.

    See 'systemctl status auditd.service' for details.

    [  OK  ] Finished Rebuild Journal Catalog.

    [  OK  ] Started RPC Bind.

             Starting Update is Completed...

    [  OK  ] Finished Update is Completed.

    You are in emergency mode. After logging in, type "journalctl -xb" to view

    system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"

    to boot into default mode.

    Give root password for maintenance

    (or press Control-D to continue): 



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------


  • 2.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Tue January 17, 2023 09:37 AM
    Edited by Tom Komadowski Tue January 17, 2023 09:41 AM
    I was able to get it working.  You are correct there is a bug in the RHEL 9 install.  I found an obscure reference in the RedHat forums about it after much searching.

    The reference page is here:  Chapter 8. Known issues Red Hat Enterprise Linux 9 | Red Hat Customer Portal

    the section is about cloning with Nutanix but it applies for PowerVC as well.  This is the section.

    Cloning or restoring RHEL 9 virtual machines that use LVM on Nutanix AHV causes non-root partitions to disappear

    When running a RHEL 9 guest operating system on a virtual machine (VM) hosted on the Nutanix AHV hypervisor, restoring the VM from a snapshot or cloning the VM currently causes non-root partitions in the VM to disappear if the guest is using Logical Volume Management (LVM). As a consequence, the following problems occur:

    • After restoring the VM from a snapshot, the VM cannot boot, and instead enters emergency mode.
    • A VM created by cloning cannot boot, and instead enters emergency mode.

    To work around these problems, do the following in emergency mode of the VM:

    1. Remove the LVM system devices file: rm /etc/lvm/devices/system.devices
    2. Recreate LVM device settings: vgimportdevices -a
    3. Reboot the VM

    This makes it possible for the cloned or restored VM to boot up correctly.



    ------------------------------
    Tom Komadowski
    ------------------------------



  • 3.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Thu January 19, 2023 02:52 AM
    Hi Tom!

    Many Thanks for the help on this! Can easily say that I would NEVER would have figured this out myself. :D 

    But like you said it works! 

    Have you tried to put those commands on cloud-init config or somewhere, to avoid the need manually push the guy up from the maintenance mode after deploy?  ...or maybe cloud-init is not run yet on that phase..so most probably it won't help..?  So should we wait for some fix from Redhat ..?  Any clue if there is any case open there already?

    Br,
    tommi

    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 4.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Fri February 24, 2023 09:58 AM

    I'm not sure if the commands would work for this.  The system crashes to emergency mode before cloud-init runs. 

    Redhat is aware of the issue but I haven't seen anything on a fix for it yet, but I haven't looked recently either.   If I find out any more info on this I will update this discussion.



    ------------------------------
    Tom Komadowski
    Principal DevOps Engineer
    Fortra, LLC.
    Eden Prairie MN
    ------------------------------



  • 5.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Mon February 27, 2023 01:18 AM

    Thanks Tom,

    Yeps, I tested it some time back and it won't work, like U said..

    Thanks for any info if U hear something regarding this.

    Br,

    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 6.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Tue February 28, 2023 03:08 AM

    It's a long shot, but can you try adding persistent_policy="by-uuid" to /etc/dracut.conf.d/policy.conf and regenerate the initramfs?

    It helped with a similar problem on SLES in the past.

    Regards,

    Levente



    ------------------------------
    Levente Szente
    ------------------------------



  • 7.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Tue February 28, 2023 03:23 AM

    Thanks Levente, I will give a try on this if no official fix comes from Redhat soon; 

    Br,

    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 8.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Tue February 28, 2023 10:18 AM

    Looks like Redhat has a workaround for this.   This is from the Redhat 9.1 release notes.

    Alternatively, to prevent the issue from occurring, do the following before cloning a VM or creating a VM snapshot:

    1. Uncomment the use_devicesfile = 0 line in the /etc/lvm/lvm.conf file
    2. Reboot the VM

    I haven't tested it yet though.



    ------------------------------
    Tom Komadowski
    Principal DevOps Engineer
    Fortra, LLC.
    Eden Prairie MN
    ------------------------------



  • 9.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Wed March 01, 2023 01:54 AM

    Hi,

    I tested it now....and it works! :)

    At least VM Clone worked OK on PowerVC 2.1 after this change, so I assume also normal Capture / Deploy image will work as well.

    Many Thanks Tom for digging this info & helping in this ! 

    Br,

    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 10.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Thu March 02, 2023 01:57 AM
    Edited by Walter Orb Thu March 02, 2023 01:58 AM

    Coming late to this discussion, but setting  use_devicesfile = 0 in  /etc/lvm/lvm.conf is what I'm using in our PowerVC landscape and it works just fine for capture and deploy.

    Unfortunately, there is still this bug to deal with:

    RHEL8/9: LPAR is not booting up after inactive LPM - Red Hat Customer Portal

    That applies to PowerVC deploy operations too. A fix for this problem has already been build, but depending on which release you are using, it's not available the official repositories yet.

    The described workaround to use a compute template with dedicated processors instead of shared processors works for PowerVC deploy operations too.



    ------------------------------
    Walter Orb

    IBM
    ------------------------------



  • 11.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Thu March 02, 2023 03:01 AM

    Thanks Walter,

    Yes. the use_devicesfile = 0 setting works, I tested it too yesterday.

    And Thanks for the info regarding the LPM bug; that was news to me!

    Any details which version (PowerVC / RHEL 8.x) are affected now? 

    And where can we find info regarding the official fix availability?

    Br,

    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 12.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    Posted Thu March 02, 2023 07:53 AM

    Hi Tommi, 

    I'm on PowerVC version 2.1 already, but the problem is an issue with grub and it shouldn't matter which PowerVC version you are using.

    I think it affects RHEL 8.6 and higher and all RHEL 9 releases. The bugzilla items do not talk about planned release dates, I think you'd have to open a support case with Red Hat if you need this info. 



    ------------------------------
    Walter Orb
    IBM Technology Engineering for SAP
    IBM
    ------------------------------



  • 13.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Thu March 02, 2023 07:58 AM

    Thanks, 

    We do have CCSP contract with Redhat covering those Support etc stuff..so I might open a case for this;

    Br,

    t



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 14.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Fri March 03, 2023 02:15 AM

    Hi,

    Case #03452076 opened.



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------



  • 15.  RE: PowerVC 2.0.3 / RHEL9 Guest VM

    IBM Champion
    Posted Wed May 17, 2023 04:10 AM

    Hi,

    This should be fixed now:
    https://access.redhat.com/errata/RHBA-2023:3044



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------