PowerVC

 View Only

  • 1.  PowerVC 2.2.1.1 / Failed to mount Failed to mount /srv/node/partition1.

    Posted Tue January 21, 2025 06:50 AM

    Hi,

    Anyone bumped into an error where srv/node/partition1 (Swift Object storage loop device) is not mounting / is somehow corrupted?

    My Dev PowerVC node (RHEL9)  is not booting up anymore, goes to Rescue Mode instead.

     systemd[1]: Mounting /srv/node/partition1...

     mount[1176]: mount: /srv/node/partition1: can't read superblock on /dev/loop0.

     systemd[1]: srv-node-partition1.mount: Mount process exited, code=exited, stat>

     systemd[1]: srv-node-partition1.mount: Failed with result 'exit-code'.

     systemd[1]: Failed to mount /srv/node/partition1.

    There if I try to mount /srv/node/partition1  manually, it gives:

    [root@power-pvc-d061 ~]# mount -va

    [  560.194503] loop0: detected capacity change from 0 to 12582912

    [  560.195961] XFS (loop0): Mounting V5 Filesystem 77333875-923d-41eb-86c8-72745e181978

    [  560.343641] XFS (loop0): Starting recovery (logdev: internal)

    [  562.340029] loop: Write error at byte offset 2507407360, length 65536.

    [  562.340042] I/O error, dev loop0, sector 4897280 op 0x1:(WRITE) flags 0x1000 phys_seg 1 prio class 2

    [  562.340054] XFS (loop0): metadata I/O error in "xfs_buf_ioend_handle_error+0x130/0x490 [xfs]" at daddr 0x4aba00 len 128 error 5

    [  562.340130] XFS (loop0): Metadata I/O Error (0x1) detected at xfs_buf_ioend_handle_error+0x290/0x490 [xfs] (fs/xfs/xfs_buf.c:1233).  Shutting down filesystem.

    [  562.340193] XFS (loop0): Please unmount the filesystem and rectify the problem(s)

    [  562.342248] XFS (loop0): log mount/recovery failed: error -5

    [  562.342438] XFS (loop0): log mount failed

    mount: /srv/node/partition1: can't read superblock on /dev/loop0.

    [root@power-pvc-d061 ~]# ls -lart /srv/node/partition1

    total 0

    drwxr-xr-x. 2 swift swift  6 Sep 24 11:09 .

    drwxr-xr-x. 4 swift swift 38 Sep 24 11:09 ..

    [root@power-pvc-d061 ~]# ls -lart /dev/loop0

    brw-rw----. 1 root disk 7, 0 Jan 21 13:27 /dev/loop0

    [root@power-pvc-d061 ~]# ls -lart /srv/node/drives/swift.img

    -rwxr-xr-x. 1 swift swift 6442450944 Jan 21 13:27 /srv/node/drives/swift.img

    [root@power-pvc-d061 ~]# cat /etc/fstab |grep swift

    /srv/node/drives/swift.img /srv/node/partition1 xfs defaults 0 0

    Any idea how to recover from this?

    Thanks,

    Br,

    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry Tech Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------


  • 2.  RE: PowerVC 2.2.1.1 / Failed to mount Failed to mount /srv/node/partition1.

    Posted Thu January 23, 2025 09:32 AM

    Hi 

    have you tried to repair the file device ?

    xfs_repair  /srv/node/drives/swift.img

    any selinux issue ?

    Regards



    ------------------------------
    Stefano Vargiu
    ------------------------------

    Original Message


  • 3.  RE: PowerVC 2.2.1.1 / Failed to mount Failed to mount /srv/node/partition1.

    Posted Thu January 23, 2025 09:32 AM

    Hi 

    have you tried to check the file device ?

    xfs_repair /srv/node/drives/swift.img

    Any selinux problem ?



    ------------------------------
    Stefano Vargiu
    ------------------------------

    Original Message


  • 4.  RE: PowerVC 2.2.1.1 / Failed to mount Failed to mount /srv/node/partition1.

    Posted Thu January 23, 2025 02:49 PM

    What does the Swift log say?

    Has the OS been raised?



    ------------------------------
    Andre Lutz
    ------------------------------

    Original Message


  • 5.  RE: PowerVC 2.2.1.1 / Failed to mount Failed to mount /srv/node/partition1.

    Posted Fri January 24, 2025 12:49 AM

    Hi all,

    Opened PMR on this & got quick help from IBM RHEL team; below action plan fixed the issue in case someone else hits the same issue:

    Action Plan: Resolving Filesystem Corruption on /srv/node/partition1

    1. Boot into Rescue Mode

    1. Reboot the system and choose Rescue Mode from the boot menu.
    2. Ensure the root filesystem is mounted in read-write mode.

    mount -o remount,rw /

    2. Verify Loop Device Setup

    1- Check if the loop device is correctly mapped to the image file:

    losetup -a

    If /dev/loop0 is not listed, attach the image file manually:

    losetup /dev/loop0 /srv/node/drives/swift.img

    2- Confirm the filesystem type of the loop device:

    blkid /dev/loop0

    3. Unmount and Inspect Filesystem

    1- Ensure /srv/node/partition1 is not mounted:

    umount /srv/node/partition1

    2- Run a filesystem check and repair using xfs_repair command:

    xfs_repair /dev/loop0

    4. Re-Mount the Filesystem

    1- Attempt to mount the filesystem:

    mount /srv/node/partition1

    2- Verify the contents using:

    ls -l /srv/node/partition1

    5. Check Disk Image and Underlying Storage

    1- Run a quick integrity check on the image file to ensure it's intact:

       - file /srv/node/drives/swift.img 
       - dd if=/srv/node/drives/swift.img bs=1M count=10 | hexdump -C

    2- If the image file appears corrupted, recreate it:

     

    Create a new empty image:

    truncate -s 6G /srv/node/drives/swift.img
    mkfs.xfs /srv/node/drives/swift.img

     

    Update /etc/fstab and mount:

    mount -a



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    Tietoevry Tech Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------

    Original Message