AIX

 View Only
Expand all | Collapse all

Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

  • 1.  Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Thu March 30, 2023 01:13 PM

    Hello everyone.

    We want to mirror the rootvg on a SAN disk, but skip the next issue:

    0516-1980 extendvg: Block size of all disks in the volume group must be the same. Cannot mix disks with different block sizes.

    Is there any way to change the disk block zise on AIX disk (/dev/hdiskX) before to extend or add the new PV to the VG and get the mirror of the rootvg ?

    Note: The disk where the OS resides is an internal NVMe disk.

      

    Regards,

    JJMO



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 2.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Thu March 30, 2023 01:44 PM
    On Thu, Mar 30, 2023 at 05:13:27PM +0000, Juan Jose Morales Ortiz via IBM Community wrote:
    > We want to mirror the rootvg on a SAN disk, but skip the next issue:
    >
    > 0516-1980 extendvg: Block size of all disks in the volume group must be the same. Cannot mix disks with different block sizes.
    >
    > Is there any way to change the disk block zise on AIX disk (/dev/hdiskX) before to extend or add the new PV to the VG and get the mirror of the rootvg ?
    >
    > Note: The disk where the OS resides is an internal NVMe disk.

    You may have to do an alt_disk_copy from the NVMe to the SAN disk. At
    least that is only a single reboot.

    Do you have an "lsattr -El" and "lscfg -vl" on both disks? Where is
    the block size identified?

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 3.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 12:15 PM

    For the mirror operation to work, the disks must have the same block size. I believe you can set the LUN's blobksize in the storage subsystem. It the storage subsystem report a different block size between the 2 disks, AIX connot override it. But the storage system can. 

    Unfortunately, I personally do now know how to change this in the storage subsystem. 



    ------------------------------
    Grover Davidson
    ------------------------------



  • 4.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 09:36 PM

    Thanks Grover, I'm just reviewing the allocation and modification at the storage level. If I find something I'll post it.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 5.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 09:31 PM

    Thank you very much Russel, I will review the comand.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 6.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Thu March 30, 2023 08:34 PM

    Hi Juan Jose

    You cant add other new volumes to an Volume Group id the block size of the new one is not equal to existing disks.
    That is a problem with new hard drives with 4KB block size when legacy and generally SAN disks use just 512B.
    Unfortunatelly this is a fixed setting coming from the hardware and it is not possible to be changed. 

    I suggest you to do an mksysb and restore it on a new disk (LUN from SAN) 

    I don't know if alt_disk_copy could work for this, I understand the process is like to a mirrorvg and then a splitvg more than a backup/restore from a VG to another. So, it should need again add the new disk to VG first... (I don't pretty sure of this, you can try it) 

    It is possible to virtualize a file from VIOS to server to be seen as a new hdisk which broke with the fixed disk block size masking a new 512B device... but I do not recommend this because of performance and access redundancy.



    ------------------------------
    Luis Alberto Rojas Kramer
    ------------------------------



  • 7.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 03:26 AM

    Hi Juan Jose,

    I advice you to read 

    https://techchannel.com/SMB/06/2020/spare-disk-values-and-challenges




    ------------------------------
    Sylvain
    ------------------------------



  • 8.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 09:40 PM

    Hello Sylvain, good reading... I´ll take the command into account. Thank you.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 9.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    IBM Champion
    Posted Fri March 31, 2023 05:09 AM

    alt_disk_copy will work (it doesn't use mirror&split, instead it uses mkvg/mklv/mkfs then copies the files to the new structure)

    As to virtualizing through the VIOS, I don't know if it shadows the block size (it should), but even if it doesn't now, it will in the future.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 10.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 05:19 AM
    On Fri, Mar 31, 2023 at 09:09:10AM +0000, Jos? Pina Coelho via IBM Community wrote:
    > alt_disk_copy will work (it doesn't use mirror&split, instead it uses mkvg/mklv/mkfs then copies the files to the new structure)
    >
    > As to virtualizing through the VIOS, I don't know if it shadows the block size (it should), but even if it doesn't now, it will in the future.

    I would like to find where you can display the block size of a PV, so
    you can plan ahead. lscfg and lsattr didn't show it.

    Have you seen a command that does?

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 11.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    IBM Champion
    Posted Fri March 31, 2023 05:56 AM
    Edited by José Pina Coelho Fri March 31, 2023 07:08 AM

    According to IBM, you need to install IBM i ... (https://www.ibm.com/support/pages/how-determine-byte-sector-size-disk-unit) - yes, just joking.

    mkvg knows about sector sizes, but calls them block size with will bring in a lot of unrelated search answers.

    # strings /etc/mkvg|sort -u|egrep "(blk)|(block)"
    bigvg: vgda_size=%u, vgda_blocks=%u
    oldvg: vgda_size=%u, vgda_blocks=%u
    validate_pvs: Invalid blk size!%s (%d)
    validate_pvs: Invalid block size! %s (%d)
    validate_pvs: Mixed blk sizes! Other disks have %d block size. %s (%d)

    Edit: seems to be related to function hd_cfg, which is internal to libodm.a 

    It's going to be hidden inside the libraries.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 12.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    IBM Champion
    Posted Fri March 31, 2023 09:39 AM
    Edited by José Pina Coelho Fri March 31, 2023 09:46 AM

    This has been bugging me way too much...

    On kdb:

    (0)> scsidisk hdisk0

    NAME            ADDRESS             STATE             CMDS_OUT  CURRBUF             LOW
    hdisk0          0xF1000C01C28C4000  0x0000000000000002  0x0000    0x0000000000000000  0x0
    struct scsidisk_diskinfo {
        struct scsidisk_diskinfo *next      = 0x0;
        struct scsidisk_diskinfo *next_open = 0x0;
        dev_t  devno    = 0x8000001300000000;
    ...
        uint cfg_block_size  = 0x200;
        uint block_size      = 0x200;
    ...

    0x200 = 512.  Now, if you have a 4K disk, check it like this to see if it shows up as 0x1000.


    Edit: Further on there is an interesting structure:

        struct scsidisk_capacity64 &capacity64 = 0xF1000C01C28C6870 {
            uint64_t lba   = 0x77FFFFF;
            int len = 0x200;
        };


    The hdisk is 61440MB according to bootinfo.

    0x77FFFFF is 125829119

    125829119*512/1024/1024 is 61439, but if you add 1 to the lba, it's 61440, so lba should be the offset of the last block and not the lba count.


    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 13.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Tue April 04, 2023 11:03 AM

    Hi,
    maybe lsdev is what you are looking for (at least it shows the 4k disks):
    # lsdev -c disk
    hdisk0 Available 00-00 NVMe 4K Flash Disk
    hdisk1 Available 03-00 NVMe 4K Flash Disk
    hdisk2 Available 00-00 NVMe Flash Disk
    hdisk3 Available 03-00 NVMe Flash Disk



    ------------------------------
    Christoph Buchmann
    ------------------------------



  • 14.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Tue April 04, 2023 11:20 AM

    The easiest way I see is using KDB:

     

    # kdb

               START              END <name>

    0000000000001000 0000000007120000 start+000FD8

    F00000002FF47600 F00000002FFE1000 __ublock+000000

    000000002FF22FF4 000000002FF22FF8 environ+000000

    000000002FF22FF8 000000002FF22FFC errno+000000

    F100110A00000000 F100110A10000000 pvproc+000000

    F100110A10000000 F100110A18000000 pvthread+000000

    read vscsi_scsi_ptrs OK, ptr = 0xF1000000C0128E90

    (0)> scsidisk hdisk0 | less

    NAME            ADDRESS             STATE             CMDS_OUT  CURRBUF             LOW

    hdisk0          0xF1000C01C06D4000  0x0000000000000002  0x0000    0x0000000000000000  0x0

    struct scsidisk_diskinfo {

        struct scsidisk_diskinfo *next      = 0x0;

        struct scsidisk_diskinfo *next_open = 0x0;

        dev_t  devno    = 0x8000001000000003;

        uchar  dev_type = 0x1;

        uchar  transport_type = 0x3;

        uchar  transport_flag = 0x0;

        uchar  prot_type = 0x0;

        uchar  desired_prot_type = 0x0;

        uchar  app_tag_owner = 0x0;

        uchar  buffer_ratio = 0x0;

        uchar  async_flag   = 0x0;

        uchar  valid_cd_modes = 0x0;

        uchar  q_control_byte = 0x4;

        ushort rw_timeout    = 0x1E;

        ushort rw_max_time   = 0x0;

        ushort queue_depth   = 0x10;

        ushort scsi_version  = 0x3;

        ushort valid_flags   = 0x0;

        uint cfg_block_size  = 0x200;

        uint block_size      = 0x200;

        uint mult_of_blksize = 0x1;

        int  max_retries     = 0x5;

        int  max_flushes     = 0x20;

        int  max_request     = 0x40000;

    ...

     

    0x200 = 512 bytes.

     

    Thanks!

    Grover Davidson
    AIX Screen Team
    Phone 512-286-9689
    grover@us.ibm.com

    Dick Stratton, who was held in the Hanoi Hilton for 2,251 days as a "prisoner at war," had taught me that a call from the field is not an interruption of the daily routine; it's the reason for the daily routine. - General James Mattis

    Teamwork is built on individual contributions, each working independently but willingly helping others when needed. It is by sharing knowledge and skills that the abilities of the individual contributors is increased, and as a result, the whole team. - Grover Davidson

     

     






  • 15.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Tue April 04, 2023 01:03 PM
    On Tue, Apr 04, 2023 at 03:03:18PM +0000, Christoph Buchmann via IBM Community wrote:
    > Hi,
    > maybe lsdev is what you are looking for (at least it shows the 4k disks):
    > # lsdev -c disk
    > hdisk0 Available 00-00 NVMe 4K Flash Disk
    > hdisk1 Available 03-00 NVMe 4K Flash Disk
    > hdisk2 Available 00-00 NVMe Flash Disk
    > hdisk3 Available 03-00 NVMe Flash Disk

    To confirm, hdisk2 and hdisk3 were customized using the NVMe Create
    Namespace with LBA selection?

    hdisk0 and hdisk1 are the default in 4k LBA mode?

    That's easy enough.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 16.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Wed April 05, 2023 04:47 AM

    Hi Russell, 
    this is correct. 
    hdisk0 = 1st namespace on nvme0 (LBA 4K)
    hdisk1 = 1st namespace on nvme1 (LPA 4K)
    hdisk2 = 2nd namespace on nvme0 (LBA 512b)
    hdisk3 = 2nd namespace on nvme1 (LBA 512b)




    ------------------------------
    Christoph Buchmann
    ------------------------------



  • 17.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 05:21 AM
    Already tested several times.

    It works perfectly.




  • 18.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    IBM Champion
    Posted Fri March 31, 2023 05:28 AM

    In that case, are you passing a PV, an LV, or an SSP volume as a VSCSI volume ?  Note that the ability to use a 4096 volume with a 512 block size may go away anytime.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 19.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 05:33 AM
    Edited by Sylvain Fri March 31, 2023 05:39 AM

    We use SAN LUN seen as PV on AIX.

    Sometimes we also use LV as vscsi volume.

    Both works well.




  • 20.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    IBM Champion
    Posted Fri March 31, 2023 07:06 AM

    SG24-8062 (IBM PowerVM Best Practices) does mention using VSCSI to allow an IBM i to access 512-byte disks when you don't have 520-byte storage, but it's from 2012.

    I can't find any reference if it maps a 512 520-byte sectors to 520 512-byte sectors (which would cause a large performance impact) or it strips/adds the 8 byte CRC to each sector doing 1:1 mapping (more likely).


    Also MIA, is any reference to 4096<->512 (P2V or V2P), but doing 512 (v) on 4096 (p) would force one read for every write that isn't an aligned multiple.  4096(v) on 512(p) isn't a performance issue, since 4096 is always an aligned multiple.  No idea where that would stand on data integrity.

    Searches for "512 520 4096 disk block" seem to always land on the PLET/ALET...



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 21.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 09:45 PM

    Hello José, I think everything points to the alt_disk_copy command. Thank you very much for your collaboration.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 22.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Fri March 31, 2023 09:38 PM

    Hi Luis Alberto, I think alt_disk_copy will be the best alternative. Thank you.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------



  • 23.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Mon April 03, 2023 09:02 AM

    Here is a link where IBM explains the situation and cannot mix 5xx and 4k byte sector drives in the same volume group.  You can have them in the same chassis; (local disks), just not the same volume group.  Have not found a workaround with a SAN.
    https://www.ibm.com/support/pages/unable-add-new-disk-volume-group-due-mismatch-disk-block-size

    We run into this a lot as one of our solutions utilizes GLVM (Geographical Logical Volume Mirroring) and in that case even though we have 2 different physical hosts we run into the same challenge as we are not able to extend the volume group due to the "Remote Disk" having different byte sector and there is no way to change this that I'm aware of.

    My question to the greater group; (sorry to hijack this thread but is related) now that IBM POWER has mostly transitioned to newer 4K byte sector format is there any way to have a SAN attached disk be in the same volume group, ideally mirrored?  I see where SAN storage are now using 4K but it seems something along the path forces it to be presented as 5xx byte sector drive; (example - https://serverfault.com/questions/643535/does-aix-support-advanced-format-4k-sector-size-san-luns.

    We seem stuck having to use local 5xx byte sector drives if we want to interoperate with SAN storage.  Please share if there is a 4K byte sector solutions as we're not using Power10 with larger NVMe drives.

    Thanks,
    Ron Arms





    ------------------------------
    Ron Arms
    ------------------------------



  • 24.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Tue April 04, 2023 11:03 AM

    Hi, 
    try the NVMe namespaces feature - you can choose the LBA format when creating one. See https://www.ibm.com/docs/en/power9?topic=device-creating-namespace-nvme

    I haven't tried to mirror the disks to a SAN disk so far, but it should work

    Another nice side effect: you can use the additional (usually more or less wasted) free disk space of your VIO server NVMe 



    ------------------------------
    Christoph Buchmann
    ------------------------------



  • 25.  RE: Problem adding a new disk from IBM storage to a volumen group, mismatch disk block size

    Posted Wed April 05, 2023 10:57 AM

    Great, I'll check it out. Thank you very much Christopher.



    ------------------------------
    Juan Jose Morales Ortiz
    ------------------------------