Informix

 View Only
Expand all | Collapse all

KAIO not enabling on a device with 4096B block size

  • 1.  KAIO not enabling on a device with 4096B block size

    Posted Fri September 15, 2023 09:35 AM
    Reply

    Hello everyone,

     

    It seems others have run into this issue recently (this year) and now I am running into it.  KAIO is not enabling on a chunk that sits on a device with a 4096 B block size.

     

    14.10.FC9WE running on Ubuntu LTS 22.04.

     

    root@informix03:/# blockdev --getbsz /dev/sdb1

    4096      (virtual server SSD disk in some cloud)

     

    2023-09-14 18:27:07.931  IBM Informix Dynamic Server Version 14.10.FC9WE

    2023-09-14 18:27:07.932  Performance Advisory: Currently IBM Informix Dynamic Server cannot

              support KAIO on the device containing  '/ilink/prod/root'  (this is on /dev/sdb1, cooked chunk, on an ext2 partition)

    2023-09-14 18:27:07.932   Results: Direct and concurrent I/O are disabled for this chunk.

    2023-09-14 18:27:07.932   Action: Standard AIO (possibly buffered) will be used for I/O

               to this chunk.

     

    I suspect there is no fix to this in the software yet (14.10.FC10 ?).  If you ran into this, what did you do?

     

    Thank you,

     

    Hal

     

    Hal Maner

    M Systems International, Inc.

    hmaner@msystemsintl.com

    www.msystemsintl.com

     



  • 2.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 04:01 AM
    Reply
    Hal,

    I am not sure this is related to the block size of the underlying disk, but there is some kind
    of defect from my point of view.
    I can reproduce this when the device has ext2 filesystem, also on ext4

    09:32:20  IBM Informix Dynamic Server Version 14.10.FC10IE
    09:32:20  Performance Advisory: Currently IBM Informix Dynamic Server cannot
              support KAIO on the device containing
              '/srv/dbstest/rootdbs'
    09:32:20   Results: Direct and concurrent I/O are disabled for this chunk.
    09:32:20   Action: Standard AIO (possibly buffered) will be used for I/O
               to this chunk.

    BUT if I directly use the block device as rootdbs (so not a file), this does not happen.
    09:39:08  IBM Informix Dynamic Server Version 14.10.FC10IE
    09:39:08  VP 1 is requesting 16384 kaio event resources.
    09:39:27  IBM Informix Dynamic Server Initialized -- Complete Disk Initialized.

    # ls -ls
    insgesamt 0
    0 lrwxrwxrwx 1 root root 24 Sep 16 09:38 rootdbs -> /dev/mapper/vg01-dbstest
    # blockdev --getbsz /dev/mapper/vg01-dbstest
    4096

    (this is a lvm mapped device on a NVME which has also 4k block size).

    So there is an issue with the filesystem approach rather than the block size from my point of view.
    I believe only IBM support can answer this.
    I tested this in Ubuntu 20.04 and got the same results when initializing a rootdbs on a ext4 fs.
    (same message as above that KAIO cannot be supported).

    Not sure if a different OS is behaving correctly.


    Best,
    MARCUS HAARMANN



    Original Message


  • 3.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 06:33 AM
    Reply
    I was curious and installed a 14.10 Dev version in a fresh Fedora 38 instance.
    Same error occurred there when initializing the rootdbs.
    So no Ubuntu issue. Kernel 6.2.9 in the Fedora, 6.2.0 on my Ubuntu 22, 5.15 in my Ubuntu 20 test.
     
    Time to file a bug I would say.
    MARCUS HAARMANN




    Original Message


  • 4.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 07:13 AM
    Reply
    Interestingly, this seems to happen only at initialization time.
    When restarting the instance, kaio is started without any error in the logs.
    Maybe that's why I did not recognize it before.
    Also onstat -g iov shows kio vps correctly (not after oninit -i, but after restart).

    Directliy after oninit -i:
    onstat -g iov
    IBM Informix Dynamic Server Version 14.10.FC10IE -- On-Line -- Up 00:00:32 -- 1227304 Kbytes
    2023-09-16 13:08:16

    AIO I/O vps:
    class/vp/id s  io/s totalops  dskread dskwrite  dskcopy  wakeups  io/wup  errors tempops
     fifo  7  0 i   0.0        0        0        0        0        1   0.0       0        0
      msc  6  0 i   0.0        1        0        0        0        2   0.5       0        1
      aio  5  0 s 289.3     8967     3287     5530        0     8588   1.0       0     1427
      pio  4  0 i   0.2        5        0        5        0        6   0.8       0        5
      lio  3  0 i  64.9     2011        0     2011        0     2012   1.0       0     2011


    After a restart (KAIOON=1000):

    13:08:32  Event notification facility epoll enabled.
    13:08:32  IBM Informix Dynamic Server Version 14.10.FC10IE
    13:08:32  VP 8 is requesting 1000 kaio event resources.
    13:08:32  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

    onstat -g iov

    IBM Informix Dynamic Server Version 14.10.FC10IE -- On-Line -- Up 00:00:16 -- 1227304 Kbytes
    2023-09-16 13:08:42

    AIO I/O vps:
    class/vp/id s  io/s totalops  dskread dskwrite  dskcopy  wakeups  io/wup  errors tempops
     fifo  7  0 i   0.0        0        0        0        0        1   0.0       0        0
      kio -1  0 i  92.5     1480      313     1167        0     2644   0.6       0        0
      kio -1  1 i  35.3      565      312      253        0     1087   0.5       0        0
      msc  6  0 i   0.2        3        0        0        0        4   0.8       0        3
      aio  5  0 i  98.9     1583     1194      226        0     1584   1.0       0        0
      pio  4  0 i   0.0        0        0        0        0        1   0.0       0        0
      lio  3  0 i   0.0        0        0        0        0        1   0.0       0        0


    Weird behaviour. 
    MARCUS HAARMANN



    Original Message


  • 5.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 12:58 PM
    Reply
    Marcus,

    Thank you very much for testing this and your feedback.

    I was hopeful that all would be well with KAIO after a restart like your case but unfortunately the same messages in online.log persist after a restart in my case and my onstat -g iov output is the same as before restart and I see no kio in that output.

    Hal Maner



    Original Message


  • 6.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 03:10 PM
    Reply
    Hmm, this is strange.
    Did you check all the prerequisites?
    Libaio needs to be installed, direct_io setting should be 1, listeners should have the NET type, and aio-max-nr needs to be bigger than KAIOON value.
    Can you test with ext4 or with direct device links?

    Sorry, no better ideas....

    Best,

    Marcus 


    Gesendet von Outlook für Android



    Original Message


  • 7.  RE: KAIO not enabling on a device with 4096B block size

    Posted Sat September 16, 2023 03:18 PM
    Reply
    Yes, all prerequisites are there.  

    No, I need to keep the file system and the chunks as is.

    Thank you,

    Hal



    Original Message


  • 8.  RE: KAIO not enabling on a device with 4096B block size

    Posted Mon September 18, 2023 08:24 AM
    Reply
    This is already discussed in this forum. At this moment KAIO is disabled with sector sizes larger than 1k

    This is due to standard page size of Informix in Linux is 2k and page size should be aligned to disk sector sizes.

    There is a PMR request to enable KAIO on dbspaces with sector sizes multiple of sector sizes (e.g. 4k, 8k, 12k or 16k)

    Vicente Salvador



    Original Message


  • 9.  RE: KAIO not enabling on a device with 4096B block size

    Posted Mon September 18, 2023 08:58 AM
    Reply

    Vicente, thank you.  it does not sound like it, but, Is this a relatively new "bug" or weakness with IDS 14? 

     

    I ask because I did not run into such a problem with the predecessor of this server that I had setup back in 2020 using 12.10.FC4 on Linux.  I checked and the device block size is also 4096 there yet KAIO has been running with basically the same configuration and onstat -g iov shows kio.

     

    Hal

     




    Original Message


  • 10.  RE: KAIO not enabling on a device with 4096B block size

    Posted Mon September 18, 2023 09:07 AM
    Reply
    Vicente,
    can you then explain to me why this is working on my system when database is restarted ?
    On initial allocation of the rootdbs with oninit -i, this does not work, resulting in the discussed warning.
    When restarting, KAIO is established without warning and I can see it in onstat -g iov.
    For Hal, this does not seem to work, maybe because the filesystem does not start a 4k boundary ?

    I am running the filesystem on a lvm based subpartition, formatted with ext2.
    Maybe my lvm setup is aligned to a 4k page boundary and works because of this ?
    Informix is setup with 2k pages here (standard, new installation).
    (file descriptor is shown in fdinfo as  0150002)
     040000 O_DIRECT
     010000 O_DSYNC
             02 RDWR
    0100000 O_LARGEFILE

    Immediately after initialization, the mode is 0110002.
    (O_DIRECT is missing)

    Thank you for your opinon.
    MARCUS HAARMANN



    Original Message


  • 11.  RE: KAIO not enabling on a device with 4096B block size

    Posted Tue September 19, 2023 03:53 AM
    Reply
    You can find a good answer from Vladimir in this post:


    Vicente Salvador

    Vicente Salvador
    Software Architect
    DEISTER S.A.

    vsalvador@deister.es

    Sant Pere Claver, 15
    Barcelona 08017 – Spain

    Phone: +34 932 063298
                +34 902 188080
    Perú:    +51 989 702786

    https://www.credly.com/users/vicente-salvador/badges





    Original Message


  • 12.  RE: KAIO not enabling on a device with 4096B block size

    Posted Tue September 19, 2023 11:20 AM
    Reply
    Hi Vicente,

    I read the posting and found that it is related to hardware rather than filesystem type.
    My device shows a 512bytes block size, which means that KAIO should be working.

    My question was, why does it not work at initialisation time ?
    I am getting the mentioned warning in the online.log, and after a restart the KAIO is
    working correctly.
    lsblk -o NAME,PHY-SEC,LOG-SEC /dev/nvme0n1p1
    NAME      PHY-SEC LOG-SEC
    nvme0n1p1     512     512

    For Hal, this is not working at all, probably because the underlying disk is a 4k block size media.
    Maybe he can check the media with lsblk in order to see what type of hardware is installed.
    I remember now that our admins tried to get hardware with smaller block sizes for the IDS systems.

    So this is understandable, knowing there is an outstanding APAR on this issue.

    On my machine, it is working in general, but not at initialization time (which is not a real issue
    probably, because typically we are restarting the engine some times until it gets productive,
    but we could maybe forget about it, resulting in a problematic instance until we are restarting).
    MARCUS HAARMANN



    Original Message