Informix

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

Trying to create root dbx chunk directly on a device, shows a better understanding message:

18:41:19  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:41:19  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/dev/md0'
          Its block size (4096) is larger than 1024.
18:41:19   Results: Direct and concurrent I/O are disabled for this chunk.
18:41:19   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.

So, it seams than informis only allows to use DIRECT/IO y block size is lower than 1024 bytes. My disks have sector size of 4096 bytes. Is there a workaround?

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

------------------------------
Vicente Salvador Cubedo
------------------------------

The direct I/O on Linux requires alignment on the block boundary, so for the devices where block size exceeds 512 (or, maybe, 1024) bytes (for example advanced format devices with no 512 byte emulation) Informix will disable direct I/O.

Since it's an "md" device - I wonder if it's possible to recreate it with 1k or 512 byte block ?

You can check block size for devices by something like:

    lsblk -o NAME,PHY-SEC,LOG-SEC

and/or

    blockdev --getss /dev/xxx

    blockdev --getpbsz /dev/xxx

    blockdev --getbsz /dev/xxx

I think the "important" one is the logical block (sector) size (getss), but I may be mistaken.

Trying to create root dbx chunk directly on a device, shows a better understanding message:

18:41:19  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:41:19  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/dev/md0'
          Its block size (4096) is larger than 1024.
18:41:19   Results: Direct and concurrent I/O are disabled for this chunk.
18:41:19   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.

So, it seams than informis only allows to use DIRECT/IO y block size is lower than 1024 bytes. My disks have sector size of 4096 bytes. Is there a workaround?

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

------------------------------
Vicente Salvador Cubedo
------------------------------

Hi Vladimir,

I've made al kind of testing and I identified the DIRECT I/O disabled by Informix in related to logical disk sector size, no matter if this disks are joined in a RAID or accessed directly or the filesystem used: ext2, ext3 or xfs

If the hard disk has a logical sector size of 4096 bytes Informix disables Direct I/O and if the hard disk has a logical sector size of 512 bytes, Direct I/O works.

So next question is, Is this related to some hardcoded value inside Informix only working with 512 bytes blocs. Modern Linux allows to use Direct I/O with other sector sizes. Software should align to the size of the disk sector, but it is possible.

https://access.redhat.com/articles/3911611

In resume:

O_DIRECT requires that I/O occur in multiples of 512 bytes and from memory aligned on a 512-byte boundary because O_DIRECT performs direct memory access (DMA) straight to the backing store, bypassing any intermediate buffers. Performing unaligned transfers would require "fixing up" the I/O by explicitly aligning the user-space buffer and zeroing out the slack (the space between the end of the buffer and the next 512 byte multiple), obviating the benefits of O_DIRECT.

Here's another way to look at it: The "beauty" of O_DIRECT (such as it is) is that it cuts out the VM from the I/O process. No copying from user to kernel-space, no page cache, no pages period. But this means all the little things that the kernel handles for you-alignment being the biggest-you, the user, now need to handle. The underlying backing store expects everything in sectors, so you need to talk in sectors, too.

That is why the magic number isn't always 512 bytes. It is the sector size of the underlying block device. This number is almost always 512 bytes for a standard hard drive, but it can vary. For example, ISO 9660, the standard format of CD-ROMs, sports 2048 byte sectors. You can use the BLKSSZGET ioctl to obtain a backing store's sector size. Portable code should use the value returned from this ioctl and not hard-code 512 bytes.

So, I think there is maybe an issue in Informix just using a hardcoded 512 value and disabling Direct I/O if sector size is not 512 instead of reading the logical sector size and using it.

Regards,

The direct I/O on Linux requires alignment on the block boundary, so for the devices where block size exceeds 512 (or, maybe, 1024) bytes (for example advanced format devices with no 512 byte emulation) Informix will disable direct I/O.

Since it's an "md" device - I wonder if it's possible to recreate it with 1k or 512 byte block ?

You can check block size for devices by something like:

    lsblk -o NAME,PHY-SEC,LOG-SEC

and/or

    blockdev --getss /dev/xxx

    blockdev --getpbsz /dev/xxx

    blockdev --getbsz /dev/xxx

I think the "important" one is the logical block (sector) size (getss), but I may be mistaken.

Trying to create root dbx chunk directly on a device, shows a better understanding message:

18:41:19  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:41:19  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/dev/md0'
          Its block size (4096) is larger than 1024.
18:41:19   Results: Direct and concurrent I/O are disabled for this chunk.
18:41:19   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.

So, it seams than informis only allows to use DIRECT/IO y block size is lower than 1024 bytes. My disks have sector size of 4096 bytes. Is there a workaround?

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

------------------------------
Vicente Salvador Cubedo
------------------------------

Hi Vladimir,

I've made al kind of testing and I identified the DIRECT I/O disabled by Informix in related to logical disk sector size, no matter if this disks are joined in a RAID or accessed directly or the filesystem used: ext2, ext3 or xfs

If the hard disk has a logical sector size of 4096 bytes Informix disables Direct I/O and if the hard disk has a logical sector size of 512 bytes, Direct I/O works.

So next question is, Is this related to some hardcoded value inside Informix only working with 512 bytes blocs. Modern Linux allows to use Direct I/O with other sector sizes. Software should align to the size of the disk sector, but it is possible.

https://access.redhat.com/articles/3911611

In resume:

O_DIRECT requires that I/O occur in multiples of 512 bytes and from memory aligned on a 512-byte boundary because O_DIRECT performs direct memory access (DMA) straight to the backing store, bypassing any intermediate buffers. Performing unaligned transfers would require "fixing up" the I/O by explicitly aligning the user-space buffer and zeroing out the slack (the space between the end of the buffer and the next 512 byte multiple), obviating the benefits of O_DIRECT.

Here's another way to look at it: The "beauty" of O_DIRECT (such as it is) is that it cuts out the VM from the I/O process. No copying from user to kernel-space, no page cache, no pages period. But this means all the little things that the kernel handles for you-alignment being the biggest-you, the user, now need to handle. The underlying backing store expects everything in sectors, so you need to talk in sectors, too.

That is why the magic number isn't always 512 bytes. It is the sector size of the underlying block device. This number is almost always 512 bytes for a standard hard drive, but it can vary. For example, ISO 9660, the standard format of CD-ROMs, sports 2048 byte sectors. You can use the BLKSSZGET ioctl to obtain a backing store's sector size. Portable code should use the value returned from this ioctl and not hard-code 512 bytes.

So, I think there is maybe an issue in Informix just using a hardcoded 512 value and disabling Direct I/O if sector size is not 512 instead of reading the logical sector size and using it.

Regards,

The direct I/O on Linux requires alignment on the block boundary, so for the devices where block size exceeds 512 (or, maybe, 1024) bytes (for example advanced format devices with no 512 byte emulation) Informix will disable direct I/O.

Since it's an "md" device - I wonder if it's possible to recreate it with 1k or 512 byte block ?

You can check block size for devices by something like:

    lsblk -o NAME,PHY-SEC,LOG-SEC

and/or

    blockdev --getss /dev/xxx

    blockdev --getpbsz /dev/xxx

    blockdev --getbsz /dev/xxx

I think the "important" one is the logical block (sector) size (getss), but I may be mistaken.

Trying to create root dbx chunk directly on a device, shows a better understanding message:

18:41:19  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:41:19  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/dev/md0'
          Its block size (4096) is larger than 1024.
18:41:19   Results: Direct and concurrent I/O are disabled for this chunk.
18:41:19   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.

So, it seams than informis only allows to use DIRECT/IO y block size is lower than 1024 bytes. My disks have sector size of 4096 bytes. Is there a workaround?

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

------------------------------
Vicente Salvador Cubedo
------------------------------

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

Vicente:

There are several things that can prevent a chunk from participating in DIRECT_IO:

• Cooked devices do not all participate.
• Some filesystems do not support direct_io
• There may not be sufficient KAIO resources allocated in the OS. Check with:
  cat /proc/sys/fs/aio-max-nr
  You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root:
  sudo echo 129734 >/proc/sys/fs/aio-max-nr

What does this chunk look like?  ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs

Art

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

It's not related to the server. It's related to the filesystem or underlaying characteristics. In same server, placing the rootdbs cooked file in one filesystem or other makes KAIO to work or not to work.

In this filesystem, KAIO works

# fdisk -l /dev/sda
Disk /dev/sda: 19.64 TiB, 21598316789760 bytes, 42184212480 sectors
Disk model: PERC H755 Front
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 650B3DF9-63AC-4B78-9129-01DEBC46A3A6
Device     Start         End     Sectors  Size Type
/dev/sda1   2048 42184210431 42184208384 19.6T Linux filesystem

# xfs_info /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=32, agsize=164782016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=5273024512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

In this filesystem KAIO does not work

# fdisk -l /dev/md0
Disk /dev/md0: 5.24 TiB, 5760724500480 bytes, 1406426880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 32768 bytes / 98304 bytes

# xfs_info /dev/md0
meta-data=/dev/md0               isize=512    agcount=32, agsize=43950832 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=1406426624, imaxpct=5
         =                       sunit=8      swidth=24 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Only diference between filesystems is sector size. But this comes from undelaying hardware (disk characteristics). Can this be considered a bug or there are some technical restrictions preventing to use disk with sector size 4096 for Informix?

Vicente:

There are several things that can prevent a chunk from participating in DIRECT_IO:

• Cooked devices do not all participate.
• Some filesystems do not support direct_io
• There may not be sufficient KAIO resources allocated in the OS. Check with:
  cat /proc/sys/fs/aio-max-nr
  You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root:
  sudo echo 129734 >/proc/sys/fs/aio-max-nr

What does this chunk look like?  ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs

Art

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

It's not related to the server. It's related to the filesystem or underlaying characteristics. In same server, placing the rootdbs cooked file in one filesystem or other makes KAIO to work or not to work.

In this filesystem, KAIO works

# fdisk -l /dev/sda
Disk /dev/sda: 19.64 TiB, 21598316789760 bytes, 42184212480 sectors
Disk model: PERC H755 Front
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 650B3DF9-63AC-4B78-9129-01DEBC46A3A6
Device     Start         End     Sectors  Size Type
/dev/sda1   2048 42184210431 42184208384 19.6T Linux filesystem

# xfs_info /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=32, agsize=164782016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=5273024512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

In this filesystem KAIO does not work

# fdisk -l /dev/md0
Disk /dev/md0: 5.24 TiB, 5760724500480 bytes, 1406426880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 32768 bytes / 98304 bytes

# xfs_info /dev/md0
meta-data=/dev/md0               isize=512    agcount=32, agsize=43950832 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=1406426624, imaxpct=5
         =                       sunit=8      swidth=24 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Only diference between filesystems is sector size. But this comes from undelaying hardware (disk characteristics). Can this be considered a bug or there are some technical restrictions preventing to use disk with sector size 4096 for Informix?

Vicente:

There are several things that can prevent a chunk from participating in DIRECT_IO:

• Cooked devices do not all participate.
• Some filesystems do not support direct_io
• There may not be sufficient KAIO resources allocated in the OS. Check with:
  cat /proc/sys/fs/aio-max-nr
  You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root:
  sudo echo 129734 >/proc/sys/fs/aio-max-nr

What does this chunk look like?  ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs

Art

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

------------------------------
Vicente Salvador Cubedo
------------------------------

It's not related to the server. It's related to the filesystem or underlaying characteristics. In same server, placing the rootdbs cooked file in one filesystem or other makes KAIO to work or not to work.

In this filesystem, KAIO works

# fdisk -l /dev/sda
Disk /dev/sda: 19.64 TiB, 21598316789760 bytes, 42184212480 sectors
Disk model: PERC H755 Front
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 650B3DF9-63AC-4B78-9129-01DEBC46A3A6
Device     Start         End     Sectors  Size Type
/dev/sda1   2048 42184210431 42184208384 19.6T Linux filesystem

# xfs_info /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=32, agsize=164782016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=5273024512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

In this filesystem KAIO does not work

# fdisk -l /dev/md0
Disk /dev/md0: 5.24 TiB, 5760724500480 bytes, 1406426880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 32768 bytes / 98304 bytes

# xfs_info /dev/md0
meta-data=/dev/md0               isize=512    agcount=32, agsize=43950832 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=1406426624, imaxpct=5
         =                       sunit=8      swidth=24 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Only diference between filesystems is sector size. But this comes from undelaying hardware (disk characteristics). Can this be considered a bug or there are some technical restrictions preventing to use disk with sector size 4096 for Informix?

Vicente:

There are several things that can prevent a chunk from participating in DIRECT_IO:

• Cooked devices do not all participate.
• Some filesystems do not support direct_io
• There may not be sufficient KAIO resources allocated in the OS. Check with:
  cat /proc/sys/fs/aio-max-nr
  You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root:
  sudo echo 129734 >/proc/sys/fs/aio-max-nr

What does this chunk look like?  ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs

Art

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)

I suspect a code change is required

4K Sector Support had to be added to DB2 -  https://www.ibm.com/docs/en/db2/11.5?topic=115-performance-enhancements -

Even with 4K Sector Support in DB2 it appears Direct I/O is not supported:

https://www.ibm.com/docs/en/db2/11.5?topic=variables-performance#r0005665__P_DB2_4K_DEVICE_SUPPORT 

The use of database managed (DMS) table spaces that are configured for direct disk (raw) access is not supported.

• The use of database managed (DMS) table spaces that are configured for direct disk (raw) access is not supported.
• The use of system managed (SMS) table spaces that are configured with NO FILESYSTEM CACHING is not supported.

It's not related to the server. It's related to the filesystem or underlaying characteristics. In same server, placing the rootdbs cooked file in one filesystem or other makes KAIO to work or not to work.

In this filesystem, KAIO works

# fdisk -l /dev/sda
Disk /dev/sda: 19.64 TiB, 21598316789760 bytes, 42184212480 sectors
Disk model: PERC H755 Front
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 650B3DF9-63AC-4B78-9129-01DEBC46A3A6
Device     Start         End     Sectors  Size Type
/dev/sda1   2048 42184210431 42184208384 19.6T Linux filesystem

# xfs_info /dev/sda1
meta-data=/dev/sda1              isize=512    agcount=32, agsize=164782016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=5273024512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

In this filesystem KAIO does not work

# fdisk -l /dev/md0
Disk /dev/md0: 5.24 TiB, 5760724500480 bytes, 1406426880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 32768 bytes / 98304 bytes

# xfs_info /dev/md0
meta-data=/dev/md0               isize=512    agcount=32, agsize=43950832 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=1406426624, imaxpct=5
         =                       sunit=8      swidth=24 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Only diference between filesystems is sector size. But this comes from undelaying hardware (disk characteristics). Can this be considered a bug or there are some technical restrictions preventing to use disk with sector size 4096 for Informix?

Vicente:

There are several things that can prevent a chunk from participating in DIRECT_IO:

• Cooked devices do not all participate.
• Some filesystems do not support direct_io
• There may not be sufficient KAIO resources allocated in the OS. Check with:
  cat /proc/sys/fs/aio-max-nr
  You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root:
  sudo echo 129734 >/proc/sys/fs/aio-max-nr

What does this chunk look like?  ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs

Art

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason:

 rpm -qa | grep libaio
libaio-0.3.111-13.el9.x86_64

ls -l $INFORMIXDIR/bin/oninit

-rwsr-sr--. 1 root informix 28388480 Mar  8 01:38 /home/informix/pdbwic2016_dev/bin/oninit

onstat -g cfg DIRECT_IO

IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes
2023-03-22 18:32:32

name                      current value
DIRECT_IO                 0x5

 mount | grep nvme
/dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24)

Also tested with XFS fileystem

And the message in online.log

18:17:56  IBM Informix Dynamic Server Version 14.10.FC10AEE
18:17:56  Performance Advisory: Currently IBM Informix Dynamic Server cannot
          support KAIO on the device containing
          '/INFORMIXDEV/pdbwic2016_dev/rootdbs'
18:17:56   Results: Direct and concurrent I/O are disabled for this chunk.
18:17:56   Action: Standard AIO (possibly buffered) will be used for I/O
           to this chunk.
18:17:56  IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.

In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work)