I thought the decision to deprecate raw devices had been reversed but some googling has discovered it has actually been removed from Linux as of kernel 5.14 (source
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.14-Dropping-RAW-Driver). In our production environment that is a long way off since even RHEL 8 offers raw support, being based on a v4 kernel. In any case it is fairly straightforward to alter how a device is accessed and retain the same path to it.
I've not done any testing but with block devices on Linux you can play around with which I/O scheduler is used (noop, deadline or cfq). It can be seen with:
cat /sys/block/<block device name>/queue/scheduler
Blogs about other database engines (Oracle, PostgreSQL) recommend the deadline scheduler. It would be interesting to know if you have played with this.
On RHEL 7 and its clone CentOS 7, raw devices are not too difficult, it is just a case of having suitable udev rules. Documentation on it is awful, although RedHat support portal, if you have access has some examples. I am pretty sure this works in RHEL 8 too.
This gives you a block device in /dev/mapper/
12-dm-permissions.rules:
ENV{DM_UUID}=="mpath-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", OWNER="informix", GROUP="informix", MODE="0660", ATTR{queue/scheduler}="deadline"
The above uses multipath but I think any suitable environment setting shown by
udevadm info /dev/<device name>
can be used to set up a device. You have something similar to this already.
60-raw.rules:
ENV{DM_UUID}=="mpath-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", RUN+="/bin/raw /dev/raw/raw1 /dev/mapper/$env{DM_NAME}"
KERNEL=="raw1", ACTION=="add", OWNER="informix", GROUP="informix", MODE="0660", SYMLINK+="informix/data_01"
The above creates a raw device /dev/raw/raw1 to your block device /dev/mapper and a friendly symlink to it at /dev/informix/data_01
Then:
udevadm trigger
Ben.
------------------------------
Benjamin Thompson
------------------------------
Original Message:
Sent: Wed October 27, 2021 10:14 AM
From: TOM GIRSCH
Subject: Informix on RHEL on VMWare - cooked vs. raw devices recommendation
I did incredibly detailed benchmarking on this around 2017, but an unfortunate VDI accident led to the loss of all of those detailed results. In my case, I was testing on standalone servers with some pretty beefy direct-attached storage (dual 12Gbit/sec fiber channel paths to disk, combo of SSD and traditional disk). In that configuration, the clear winner was RAID-1, KAIO off, DIRECT_IO on. I remember being surprised that turning KAIO off improved performance (by something like 5% if memory serves), but that may be a function of the fact that I'm directly referencing the block special devices. I mainly used the AGS Benchmark Tool to do the testing, but also did some testing with large batch jobs.
It's also been several years since I've performance tested direct-to-block-special against raw devices, but the last time I did, the performance difference was negligible, and let's face it, raw devices are a _huge_ PITA in Linux. And they keep trying to deprecate support anyway. Even if there is a small performance penalty, the vastly easier management makes it a price I'm willing to pay.
Final note: Although all of my performance testing has been done on standalone machines, I've also been using the direct-to-block approach on VMs (including VMWare, ESXi and Acropolis) for years with no issues.
------------------------------
TOM GIRSCH
Original Message:
Sent: Wed October 27, 2021 06:19 AM
From: Benjamin Thompson
Subject: Informix on RHEL on VMWare - cooked vs. raw devices recommendation
Hi Tom,
Interesting. I know your configuration can work reliably but is this faster than a character device and raw (with KAIO)? My understanding is a block device set up like the above will use the file system cache.
On KAIO, I recently did some testing around disabling AIO VPs and reverting to KAIO on Linux. Here were my results which were done on multiprocessor server grade kit with fibre channel connected solid state storage. Raw (character) devices are used. The load was a real application with synthetic load injectors and these have received a lot of work from our testing team to avoid problems like thrashing a small number of test accounts and a myriad of other difficulties around generating realistic load.
Run id. | Description | Approx. time |
---|
1 | Original config with KAIO | 16.00 - 16.30 |
2 | KAIO with 80000 slots | 16.30 - 16.45 |
3 | 128 AIO VPs | 16.45 - 16.57 |
I wrote a script to analyse results and the average times per row were:
Test | Average times per row (s) |
---|
Original config | 0.002003 |
KAIO 80000 | 0.001636 |
AIO VPs | 0.001937 |
I don't seem to have recorded what the original KAIO config was but at a guess it was probably around 20000 slots per CPU VP. Our issue was that the slots are divided between CPU VPs so KAIOON must be set to how many slots you want multiplied by CPU VPs.
In my results AIO VPs can be faster than an untuned KAIO set up.
My conclusion was don't go to AIO VPs, increase KAIOON and the related kernel parameter aio-max-nr if necessary. This kernel parameter can be set quite high (4 to 8 million range or even higher) as it only really exists to protect against a ridiculous number of requests.
I am not trying to invalidate your result. It is possible that on a system with not many CPU VPs, simply having more processes (AIO VPs) will improve IO rates. On a system with many processors KAIO slots may get spread too thinly across CPU VPs as I found.
Going a bit further into the problem, there is no metric in Informix to see how many KAIO requests are in flight or when you might be starved of KAIO "slots" or any obvious flags to look for. This means even knowing this is a problem is difficult to spot when most of the time your system is probably working fine.
Anyway, I hope this is interesting and it gave me an opportunity to share some performance testing results, which can be hard to obtain.
Ben.
------------------------------
Benjamin Thompson
Original Message:
Sent: Tue October 26, 2021 09:56 AM
From: TOM GIRSCH
Subject: Informix on RHEL on VMWare - cooked vs. raw devices recommendation
On CentOS (an RHEL variant) I've taken a "third way," which started off as a mistake but one that has worked well and I've continued to use: I have symbolic links pointed directly at the block special devices for the volumes I create for dbspaces/chunks. I don't create a filesystem or create chunk files; I just make a logical volume in the size that I need and I point to that. I've been running this way for at least five years without any issues. I had to do some udev rules to make sure the permissions are set correctly after a reboot, but all works very well. In our configuration, at least, we got the best performance by enabling DIRECT_IO and disabling KAIO. I did a lot of stress testing on various combinations, and that worked best. Here's what the chunk link looks like on one of my systems:
$ ls -la /informix/LINKS/ids_chaos/*root*lrwxrwxrwx. 1 informix informix 33 Jul 8 2019 /informix/LINKS/ids_chaos/chaos_root -> /dev/mapper/informixvg-chaos_root$ ls -laL /informix/LINKS/ids_chaos/*root*brw-rw----. 1 informix informix 253, 2 Oct 26 13:52 /informix/LINKS/ids_chaos/chaos_root
And here's the udev rule I created to make sure the permissions are correct (note that I'm using the volume group name as a filter):
$ cat /etc/udev/rules.d/92-local-informix-permissions.rulesENV{DM_VG_NAME}=="informixvg", ENV{DM_LV_NAME}=="*", ACTION=="add|change", MODE="0660", OWNER="informix", GROUP="informix", PROGRAM="/bin/logger /dev/$env{DM_NAME} changed to informix:informix 0660"
------------------------------
TOM GIRSCH
Original Message:
Sent: Mon October 25, 2021 07:52 AM
From: Milan Rafaj
Subject: Informix on RHEL on VMWare - cooked vs. raw devices recommendation
Hello, I need some advice what is better solution for Informix 12.10 on RHEL 7 in VMWare VM - cooked file or raw devices? From performance point of view?
Thanks a lot.
------------------------------
Milan Rafaj
Infrastructure Architects and Specialists Team Leader
Kyndryl
+420737264248
------------------------------
#Informix