Informix

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------

John

Traditionally informix would store its data on RAW devices and would communicate directly with the storage device bypassing the O/s's file management and filesystem caching layer. This is generally considered to provide the best performance.

Where your Instance is built using 'cooked' chunks, Informix provides a DIRECT_IO $ONCONFIG parameter designed to mimic the behaviour and performance of RAW device access when informix is accessing data stored in those 'cooked' chunks.

By default most flavours of Linux have KAIO switched on, so let's assume that is the case.

Informix implements KAIO by running a KIO thread per CPU VP. The KIO thread performs I/O by making system calls to the O/s, which performs the I/O independently of the CPU VP.

If the 6th position of the 'flags' column is set to 'D', in your 'onstat -d' output, DIRECT_IO was enabled when those chunks were created.

Your KIO threads should show activity in your 'onstat -g ioq' output.

From an O/s perspective, if you ascertain the PID of your Master oninit process and use if to look at the Open File Descriptors like this...

$ lsof +fg /opt/informix/data/dbspaces/*| egrep "23288|FD"

COMMAND     PID   USER   FD  TYPE        FILE-FLAG     DEVICE     SIZE/OFF NODE   NAME
oninit           23288 informix 256u  REG  RW,SYN,DIR,LG      253,6  2147483648        68  /opt/informix/data/dbspaces/rootdbs

...where my Master oninit process PID is 23288, the 'DIR' flag under the 'FILE-FLAG' column tells me that KAIO is being used.

If you are using RAW devices only, then I suspect KAIO will be enabled at the kernel level and DIRECT_IO becomes irrelevant for you. 



------------------------------
Kirit Rana
------------------------------

Original Message:
Sent: Mon January 04, 2021 05:03 AM
From: John Smith
Subject: DIRECT_IO/KAIO

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------

​Thank you for your replay

so which is better ?

Direct_IO  ; Informix manages the IOs
or
KAIO  : The OS manages IOs
?

:)

------------------------------
John Smith
------------------------------

Original Message:
Sent: Mon January 04, 2021 07:09 AM
From: Kirit Rana
Subject: DIRECT_IO/KAIO

John

Traditionally informix would store its data on RAW devices and would communicate directly with the storage device bypassing the O/s's file management and filesystem caching layer. This is generally considered to provide the best performance.

Where your Instance is built using 'cooked' chunks, Informix provides a DIRECT_IO $ONCONFIG parameter designed to mimic the behaviour and performance of RAW device access when informix is accessing data stored in those 'cooked' chunks.

By default most flavours of Linux have KAIO switched on, so let's assume that is the case.

Informix implements KAIO by running a KIO thread per CPU VP. The KIO thread performs I/O by making system calls to the O/s, which performs the I/O independently of the CPU VP.

If the 6th position of the 'flags' column is set to 'D', in your 'onstat -d' output, DIRECT_IO was enabled when those chunks were created.

Your KIO threads should show activity in your 'onstat -g ioq' output.

From an O/s perspective, if you ascertain the PID of your Master oninit process and use if to look at the Open File Descriptors like this...

$ lsof +fg /opt/informix/data/dbspaces/*| egrep "23288|FD"

COMMAND     PID   USER   FD  TYPE        FILE-FLAG     DEVICE     SIZE/OFF NODE   NAME
oninit           23288 informix 256u  REG  RW,SYN,DIR,LG      253,6  2147483648        68  /opt/informix/data/dbspaces/rootdbs

...where my Master oninit process PID is 23288, the 'DIR' flag under the 'FILE-FLAG' column tells me that KAIO is being used.

If you are using RAW devices only, then I suspect KAIO will be enabled at the kernel level and DIRECT_IO becomes irrelevant for you. 



------------------------------
Kirit Rana

Original Message:
Sent: Mon January 04, 2021 05:03 AM
From: John Smith
Subject: DIRECT_IO/KAIO

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------

RAW devices use KAIO
Cooked chunks use DIRECT_IO ensuring KAIO is enabled at the kernel layer.

If you have a HDR setup and have used ifxclone to build your Secondaries, be aware that ifxclone will disable DIRECT_IO on the Secondary and you will need to go and cleanup after ifxclone does it's magic. 

See Art's response as well as he would have a deeper understanding of the internals.

------------------------------
Kirit Rana
------------------------------

Original Message:
Sent: Mon January 04, 2021 09:01 AM
From: John Smith
Subject: DIRECT_IO/KAIO

​Thank you for your replay

so which is better ?

Direct_IO  ; Informix manages the IOs
or
KAIO  : The OS manages IOs
?

:)

------------------------------
John Smith

Original Message:
Sent: Mon January 04, 2021 07:09 AM
From: Kirit Rana
Subject: DIRECT_IO/KAIO

John

Traditionally informix would store its data on RAW devices and would communicate directly with the storage device bypassing the O/s's file management and filesystem caching layer. This is generally considered to provide the best performance.

Where your Instance is built using 'cooked' chunks, Informix provides a DIRECT_IO $ONCONFIG parameter designed to mimic the behaviour and performance of RAW device access when informix is accessing data stored in those 'cooked' chunks.

By default most flavours of Linux have KAIO switched on, so let's assume that is the case.

Informix implements KAIO by running a KIO thread per CPU VP. The KIO thread performs I/O by making system calls to the O/s, which performs the I/O independently of the CPU VP.

If the 6th position of the 'flags' column is set to 'D', in your 'onstat -d' output, DIRECT_IO was enabled when those chunks were created.

Your KIO threads should show activity in your 'onstat -g ioq' output.

From an O/s perspective, if you ascertain the PID of your Master oninit process and use if to look at the Open File Descriptors like this...

$ lsof +fg /opt/informix/data/dbspaces/*| egrep "23288|FD"

COMMAND     PID   USER   FD  TYPE        FILE-FLAG     DEVICE     SIZE/OFF NODE   NAME
oninit           23288 informix 256u  REG  RW,SYN,DIR,LG      253,6  2147483648        68  /opt/informix/data/dbspaces/rootdbs

...where my Master oninit process PID is 23288, the 'DIR' flag under the 'FILE-FLAG' column tells me that KAIO is being used.

If you are using RAW devices only, then I suspect KAIO will be enabled at the kernel level and DIRECT_IO becomes irrelevant for you. 



------------------------------
Kirit Rana

Original Message:
Sent: Mon January 04, 2021 05:03 AM
From: John Smith
Subject: DIRECT_IO/KAIO

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------

John:

Using DIRECT_IO will indeed use KAIO! Here's the full story:

When Informix accesses RAW chunks (so character devices) it uses the OS's KAIO facility by starting a KAIO thread in each CPU VP to handle all IO to/from those RAW chunks. By default, when Informix accesses chunks built using COOKED devices or filesystem COOKED files, it does not use KAIO threads but rather assigns IO requests for these chunks to AIO VPs. Such IO goes through the OS's buffer caching system. Since using the OS's lazy cache write-back system risks data loss, those COOKED chunks are opened with the O_SYNC flag enabled which forces writes to be flushed immediately from the OS cache to disk and each such write system calls will wait for the flush to complete. The extra overhead of the copy from Informix shared memory to the OS cache and then from the cache to disk results in performance that is between 15% and 40% slower than RAW device IO (typically about 25% on most systems more if there is heavy IO activity not related to Informix and if there are not enough AIO VPs). 

To mitigate this problem, and in recognition of efforts to eliminate RAW device support from Linux distributions over the years, Informix implemented DIRECT_IO for chunks built over filesystem files (but not for COOKED aka block device chunks). What this does is to open the file with the O_DIRECT flag enabled which causes IOs to bypass the OS cache entirely and write directly to storage. There is still some overhead because filesystem calls are used instead of lower level system calls, but there is not delay caused by copying the data to cache and waiting for a flush to complete. There is just a direct write from Informix cache to storage. In addition, OS KAIO functions support O_DIRECT so all reads and writes can be asynchronous without the intervention of the AIO VPs and so KAIO and KAIO threads in the CPU VPs are used for all IO activity on these chunks. The only exception is temp dbspaces which do not use DIRECT_IO by default (though you can enable that as well if it improves performance by setting DIRECT_IO to 5 (adding 0x04 to the 0x01 flag to enable direct io). This last is not documented in the online manuals, but is noted in the v14.10 onconfig.std file. 


------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
------------------------------

Original Message:
Sent: Mon January 04, 2021 05:03 AM
From: John Smith
Subject: DIRECT_IO/KAIO

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------

Oh, forgot, the overhead of using cooked file chunks with DIRECT_IO enabled is about 5% versus using RAW chunks which isn't bad. Remember to tune the OS's KAIO tunables though for best performance. The default KAIO resource settings are low.

------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com
------------------------------

Original Message:
Sent: Mon January 04, 2021 09:07 AM
From: Art Kagel
Subject: DIRECT_IO/KAIO

John:

Using DIRECT_IO will indeed use KAIO! Here's the full story:

When Informix accesses RAW chunks (so character devices) it uses the OS's KAIO facility by starting a KAIO thread in each CPU VP to handle all IO to/from those RAW chunks. By default, when Informix accesses chunks built using COOKED devices or filesystem COOKED files, it does not use KAIO threads but rather assigns IO requests for these chunks to AIO VPs. Such IO goes through the OS's buffer caching system. Since using the OS's lazy cache write-back system risks data loss, those COOKED chunks are opened with the O_SYNC flag enabled which forces writes to be flushed immediately from the OS cache to disk and each such write system calls will wait for the flush to complete. The extra overhead of the copy from Informix shared memory to the OS cache and then from the cache to disk results in performance that is between 15% and 40% slower than RAW device IO (typically about 25% on most systems more if there is heavy IO activity not related to Informix and if there are not enough AIO VPs). 

To mitigate this problem, and in recognition of efforts to eliminate RAW device support from Linux distributions over the years, Informix implemented DIRECT_IO for chunks built over filesystem files (but not for COOKED aka block device chunks). What this does is to open the file with the O_DIRECT flag enabled which causes IOs to bypass the OS cache entirely and write directly to storage. There is still some overhead because filesystem calls are used instead of lower level system calls, but there is not delay caused by copying the data to cache and waiting for a flush to complete. There is just a direct write from Informix cache to storage. In addition, OS KAIO functions support O_DIRECT so all reads and writes can be asynchronous without the intervention of the AIO VPs and so KAIO and KAIO threads in the CPU VPs are used for all IO activity on these chunks. The only exception is temp dbspaces which do not use DIRECT_IO by default (though you can enable that as well if it improves performance by setting DIRECT_IO to 5 (adding 0x04 to the 0x01 flag to enable direct io). This last is not documented in the online manuals, but is noted in the v14.10 onconfig.std file. 


------------------------------
Art S. Kagel, President and Principal Consultant
ASK Database Management Corp.
www.askdbmgt.com

Original Message:
Sent: Mon January 04, 2021 05:03 AM
From: John Smith
Subject: DIRECT_IO/KAIO

Hi

Please give me an abstract of using these two parameters, on Linux and AIX

What i know is that DIRECT_IO is recommended for coocked files (do not use KAIO) , but what about raw devices ?

Thanks in advance for your knowledges

------------------------------
John Smith
------------------------------