Informix

 View Only
Expand all | Collapse all

Informix on RHEL on VMWare - cooked vs. raw devices recommendation

  • 1.  Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Mon October 25, 2021 10:42 AM
    Hello, I need some advice what is better solution for Informix 12.10 on RHEL 7 in VMWare VM - cooked file or raw devices? From performance point of view?
    Thanks  a lot.

    ------------------------------
    Milan Rafaj
    Infrastructure Architects and Specialists Team Leader
    Kyndryl
    +420737264248
    ------------------------------

    #Informix


  • 2.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Mon October 25, 2021 11:08 AM
    Milan:

    In general, if you have DIRECT_IO enabled, the performance difference between cooked filesystem files and raw devices is about 5%. Add to that the fact that the Linux folks keep trying to get rid of raw devices, and most of us are using cooked files on Linux.

    Art

    ------------------------------
    Art S. Kagel, President and Principal Consultant
    ASK Database Management Corp.
    www.askdbmgt.com
    ------------------------------



  • 3.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Mon October 25, 2021 11:18 AM
    Hello Art, thank you for a tip.

    ------------------------------
    Milan Rafaj
    Infrastructure Architects and Specialists Team Leader
    Kyndryl
    +420737264248
    ------------------------------



  • 4.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Tue October 26, 2021 09:56 AM
    Edited by System Fri January 20, 2023 04:49 PM

    On CentOS (an RHEL variant) I've taken a "third way," which started off as a mistake but one that has worked well and I've continued to use: I have symbolic links pointed directly at the block special devices for the volumes I create for dbspaces/chunks. I don't create a filesystem or create chunk files; I just make a logical volume in the size that I need and I point to that. I've been running this way for at least five years without any issues. I had to do some udev rules to make sure the permissions are set correctly after a reboot, but all works very well. In our configuration, at least, we got the best performance by enabling DIRECT_IO and disabling KAIO. I did a lot of stress testing on various combinations, and that worked best. Here's what the chunk link looks like on one of my systems:

    $ ls -la /informix/LINKS/ids_chaos/*root*
    lrwxrwxrwx. 1 informix informix 33 Jul  8  2019 /informix/LINKS/ids_chaos/chaos_root -> /dev/mapper/informixvg-chaos_root
    $ ls -laL /informix/LINKS/ids_chaos/*root*
    brw-rw----. 1 informix informix 253, 2 Oct 26 13:52 /informix/LINKS/ids_chaos/chaos_root
    

    And here's the udev rule I created to make sure the permissions are correct (note that I'm using the volume group name as a filter):

    $ cat /etc/udev/rules.d/92-local-informix-permissions.rules
    ENV{DM_VG_NAME}=="informixvg", ENV{DM_LV_NAME}=="*", ACTION=="add|change", MODE="0660", OWNER="informix", GROUP="informix", PROGRAM="/bin/logger /dev/$env{DM_NAME} changed to informix:informix 0660"


    ------------------------------
    TOM GIRSCH
    ------------------------------



  • 5.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Wed October 27, 2021 06:19 AM
    Hi Tom,

    Interesting. I know your configuration can work reliably but is this faster than a character device and raw (with KAIO)? My understanding is a block device set up like the above will use the file system cache.

    On KAIO, I recently did some testing around disabling AIO VPs and reverting to KAIO on Linux. Here were my results which were done on multiprocessor server grade kit with fibre channel connected solid state storage. Raw (character) devices are used. The load was a real application with synthetic load injectors and these have received a lot of work from our testing team to avoid problems like thrashing a small number of test accounts and a myriad of other difficulties around generating realistic load.

    Run id. Description Approx. time
    1 Original config with KAIO 16.00 - 16.30
    2 KAIO with 80000 slots 16.30 - 16.45
    3 128 AIO VPs 16.45 - 16.57


    I wrote a script to analyse results and the average times per row were:

    Test Average times per row (s)
    Original config 0.002003
    KAIO 80000 0.001636
    AIO VPs 0.001937

    I don't seem to have recorded what the original KAIO config was but at a guess it was probably around 20000 slots per CPU VP. Our issue was that the slots are divided between CPU VPs so KAIOON must be set to how many slots you want multiplied by CPU VPs.

    In my results AIO VPs can be faster than an untuned KAIO set up.

    My conclusion was don't go to AIO VPs, increase KAIOON and the related kernel parameter aio-max-nr if necessary. This kernel parameter can be set quite high (4 to 8 million range or even higher) as it only really exists to protect against a ridiculous number of requests.

    I am not trying to invalidate your result. It is possible that on a system with not many CPU VPs, simply having more processes (AIO VPs) will improve IO rates. On a system with many processors KAIO slots may get spread too thinly across CPU VPs as I found.

    Going a bit further into the problem, there is no metric in Informix to see how many KAIO requests are in flight or when you might be starved of KAIO "slots" or any obvious flags to look for. This means even knowing this is a problem is difficult to spot when most of the time your system is probably working fine.

    Anyway, I hope this is interesting and it gave me an opportunity to share some performance testing results, which can be hard to obtain.

    Ben.


    ------------------------------
    Benjamin Thompson
    ------------------------------



  • 6.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Wed October 27, 2021 06:24 AM
    Just to add a quick note as the thread is about VMWare, I am not sure this would make much difference to the result.

    ------------------------------
    Benjamin Thompson
    ------------------------------



  • 7.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Wed October 27, 2021 10:15 AM
    I did incredibly detailed benchmarking on this around 2017, but an unfortunate VDI accident led to the loss of all of those detailed results. In my case, I was testing on standalone servers with some pretty beefy direct-attached storage (dual 12Gbit/sec fiber channel paths to disk, combo of SSD and traditional disk). In that configuration, the clear winner was RAID-1, KAIO off, DIRECT_IO on. I remember being surprised that turning KAIO off improved performance (by something like 5% if memory serves), but that may be a function of the fact that I'm directly referencing the block special devices. I mainly used the AGS Benchmark Tool to do the testing, but also did some testing with large batch jobs.

    It's also been several years since I've performance tested direct-to-block-special against raw devices, but the last time I did, the performance difference was negligible, and let's face it, raw devices are a _huge_ PITA in Linux. And they keep trying to deprecate support anyway. Even if there is a small performance penalty, the vastly easier management makes it a price I'm willing to pay.

    Final note: Although all of my performance testing has been done on standalone machines, I've also been using the direct-to-block approach on VMs (including VMWare, ESXi and Acropolis) for years with no issues.

    ------------------------------
    TOM GIRSCH
    ------------------------------



  • 8.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Wed October 27, 2021 10:49 AM
    Thank you to all for tips and experience. I will give some more info on current HW/SW configuration configuration for your information:
     ESXi servers are attached to disk array via SAN and FC-SW protocol. Each ESXi has 2 16Gbs HBA, disk array has 2 controllers attached redundantly using 4 8Gbs HBA. Informix VMDk file for chunks is created on dedicated volume from disk array LUN created from RAID-10 from SSD disks dedicated for purpose of Informix chunks only.
    VMDk si attached to VM and used whole for filesystem, where Informix chunks are created as cooked files.
    So question might be to replace the filesystem via group of blocked devices one for each chunk 


    ------------------------------
    Milan Rafaj
    Infrastructure Architects and Specialists Team Leader
    Kyndryl
    +420737264248
    ------------------------------



  • 9.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Wed October 27, 2021 11:11 AM
    If you've already got it all set up, I don't know that it's worth the effort to replace the cooked files with links to block devices. Unless there's free space on the RAID-10, you'd have to shrink or dismantle the file system, create new LUNs and incorporate those into LVM for use. That's a lot of work.

    ------------------------------
    TOM GIRSCH
    ------------------------------



  • 10.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Wed October 27, 2021 11:18 AM
    Hello Tom,
    this current status. New VM is planned to be created from the scratch with RHEL 7.6

    ------------------------------
    Milan Rafaj
    Infrastructure Architects and Specialists Team Leader
    Kyndryl
    +420737264248
    ------------------------------



  • 11.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Wed October 27, 2021 11:23 AM
    It's personal preference, but I'd make the RAID-10 visible to the OS as a single device, encapsulate it into LVM with pvcreate, and then create a volume group that holds the whole thing. Then use lvcreate to make one logical volume per chunk and link to the /dev/mapper reference directly without creating a filesystem or individual files, and any of the overhead that comes with all of that.

    Even better, if you can get it, do a couple/few smaller RAID-10s or even a bunch of RAID-1 pairs and make a physical volume for each. That will allow you to enforce some separation (e.g., keep the physical log away from the logical logs, keep the temp spaces on separate devices, etc.).

    ------------------------------
    TOM GIRSCH
    ------------------------------



  • 12.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    Posted Wed October 27, 2021 10:55 AM
    I thought the decision to deprecate raw devices had been reversed but some googling has discovered it has actually been removed from Linux as of kernel 5.14 (source https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.14-Dropping-RAW-Driver). In our production environment that is a long way off since even RHEL 8 offers raw support, being based on a v4 kernel. In any case it is fairly straightforward to alter how a device is accessed and retain the same path to it.

    I've not done any testing but with block devices on Linux you can play around with which I/O scheduler is used (noop, deadline or cfq). It can be seen with:

    cat /sys/block/<block device name>/queue/scheduler

    Blogs about other database engines (Oracle, PostgreSQL) recommend the deadline scheduler. It would be interesting to know if you have played with this.

    On RHEL 7 and its clone CentOS 7, raw devices are not too difficult, it is just a case of having suitable udev rules. Documentation on it is awful, although RedHat support portal, if you have access has some examples. I am pretty sure this works in RHEL 8 too.

    This gives you a block device in /dev/mapper/

    12-dm-permissions.rules:

    ENV{DM_UUID}=="mpath-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", OWNER="informix", GROUP="informix", MODE="0660", ATTR{queue/scheduler}="deadline"

    The above uses multipath but I think any suitable environment setting shown by

    udevadm info /dev/<device name>

    can be used to set up a device. You have something similar to this already.

    60-raw.rules:

    ENV{DM_UUID}=="mpath-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", RUN+="/bin/raw /dev/raw/raw1 /dev/mapper/$env{DM_NAME}"

    KERNEL=="raw1", ACTION=="add", OWNER="informix", GROUP="informix", MODE="0660", SYMLINK+="informix/data_01"

    The above creates a raw device /dev/raw/raw1 to your block device /dev/mapper and a friendly symlink to it at /dev/informix/data_01

    Then:
    udevadm trigger

    Ben.



    ------------------------------
    Benjamin Thompson
    ------------------------------



  • 13.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Wed October 27, 2021 11:20 AM

    It was on CentOS 7 where I discovered that raw devices were more trouble than they're worth. And because they add the extra layer of abstraction, it makes tracing problems a challenge. Instead of having a relatively straightforward /dev/[vg]/[volume] (which, of course, is referenced via symbolic link to /dev/mapper, which is itself a symbolic link to the 'real' device), I have the completely useless /dev/raw/raw[n], and you have to create and maintain mapping tables to make sense of what's what. The udev rules help a _little_, but if you want to keep everything neat and tidy, you have to create a udev rule every time you add a new chunk. That might not be a big deal when you've only got 10 or 20 chunks to keep track of, but I have, on one system alone, 386. I can easily determine what device a chunk lives on with lvdisplay -m; you've got digging to do in order to do that with raw devices.

    I suppose it just depends on what sort of pain you're willing to live with.

    [My udev rules are simple: Everything in the volume group gets informix:informix 660 permissions, and that's it; no changes to the udev file except when adding a new volume group, which is rare on our systems.]



    ------------------------------
    TOM GIRSCH
    ------------------------------



  • 14.  RE: Informix on RHEL on VMWare - cooked vs. raw devices recommendation

    IBM Champion
    Posted Wed October 27, 2021 11:27 AM
    I checked my systems and they are using deadline. But that's a good tip; it's not something I'd ever thought to check.

    ------------------------------
    TOM GIRSCH
    ------------------------------