Informix

nested-group-icon.png

DB2

Expand all | Collapse all

RAID6?

  • 1.  RAID6?

    Posted 30 days ago
    I'm lazy.  So, when I make a decision on some things, I stop making explicit efforts to continue following or researching it.

    An example of this is the whole RAID thing.  Years ago, I read much of Art's material, and decided that RAID10 was "it" for the systems that I am responsible for.  So, I went forward thinking that RAID levels were something I didn't have to agonize over any more.  "Settled science", as it were.  :)

    Imagine my mild surprise when, in a recent support case with Informix Tech Support, in which I was asking about the safety of using Solaris SVM for Informix mirroring, and, in the final case update, they suggested using RAID 6.

    (After reading Oracle's "WARNING" near the bottom of the man pages for "metainit', in which they caution about the susceptibility of SVM to write-on-write situations, I opened a priority 4 tech support case to inquire whether using SVM mirrors (configured as RAID10) was a potential risk.)

    After several months, they concluded that the answer really depended on Oracle's envisioned use case, and Informix was unable to give a definitive answer without more detail about Oracle's use case that motivated the warning.

    But, they did suggest that a good solution would be, first, external SAN storage; and, second (if external storage was not available) using internal drives configured as RAID6.

    They also suggested, as a possible third choice, ZFS, but cautioned that it was resource intensive, and might not be suitable to use on the same system as Informix.

    What caught my attention was their recommendation to use RAID-6.

    Has the world changed?  Do I need to revisit and reinvestigate this whole RAID thing (which I thought I had settled in my own mind years ago)?

    I don't want to be one of "those guys" who, being resistant to change and new knowledge, are forever living in a static past.

    Curious,

    DG



    ------------------------------
    David Grove
    ------------------------------


  • 2.  RE: RAID6?

    Posted 30 days ago

    Well, since I probably was responsible for mentioning (among other things, like ZFS) a possibility of using Informix with RAID-6, let me try to explain why.

    I am somewhat familiar with the case under discussion, but maybe not with the fine points, however the primary recommendation was to use SAN with multipathing - if needed. Granted, setting up SAN would require specific expertise and  extra hardware, but that would be by far most reliable and flexible solution. 

    I do understand the attractiveness of using  storage local to the system with 2 controllers and setting up mirror or RAID10 using disks attached to different controllers to remove an extra single point of failure (controller), however even in that configuration - system still remains a single point of failure.

    Without going into long argument about storage performance of mirror vs RAID-6, including the fact that I/O patterns for mirror are expected to be more favorable to underlying storage devices, in practical terms - with good modern controller (battery backup + reasonably sized write cache + it is enabled) most database workloads, aside from those specifically targeting I/O, will not cross threshold where difference in I/O performance between mirror and RAID-6 will be decisive factor of choosing one solution over the other.

    I'd go as far as to say that with current CPU speeds the optimization direction should be more in I/O avoidance whenever it's possible.

    If the (older) system has only 2 disks - clearly RAID-6 is not a viable option. With 6 disks and decent RAID controller - I would personally go with RAID-6, unless disks are so small that mirror can be recovered under an hour after disk failure. When one disk in mirror fails - the clock is ticking, as there is no redundancy, and despite the fact that (in theory) recovery includes only read from remaining disk and write to the new one - if for any reason remaining disk fails - the data is pretty much gone.

    As usual, the decision to use any type of RAID depends on specifics, but for multi-terabyte arrays I personally would feel safer with extra parity which RAID-6 provides.

    Now about performance. Few years ago I was evaluating IBM's FS900 flash storage with several modules which were in RAID-5.  Informix was driving close to million 4k IOPS with it (sustained!) - despite all the negative things we know about this type of RAID. Can anyone make an argument - with 8 flash modules - to use mirror / RAID10 instead ?  Both the  IOPS  and bandwidth limits at that time were from system CPU and FC adapters, not due to the fact that RAID-5 was performing worse than mirror.

    Even with locally attached SSDs - I am fairly sure that performance of RAID-6 will be adequate and provide more storage capacity, so that considering mirror solution would be unreasonable waste of resources.

    With small number of local disks capable of 150 IOPS each and write cache disabled - both mirror and RAID-6 will give awful performance with mirror being slightly less awful, but it's already 21st century - do we have to suffer needlessly? 



    ------------------------------
    Vladimir Kolobrodov
    ------------------------------



  • 3.  RE: RAID6?

    Posted 29 days ago
    Vladimir:

    You clearly haven't read my paper on why RAID5 and similar schemes such as RAID6 are not safe for important data. Here's a link for your edification:


    I am not alone in this insistence on avoiding RAID5 or RAID6.

    Art

    Art S. Kagel, President and Principal Consultant
    ASK Database Management


    Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.








  • 4.  RE: RAID6?

    Posted 29 days ago
    Hi, Art,

    as I've said, I do not want to start a discussion about theoretical advantages and shortcomings of the various redundant storage techniques.

    In tens of years of personal use of RAID-6 both with Informix and for multi-terabyte storage systems - I have yet to encounter a single case of data corruption, and believe me - I am an adept in creating, keeping, using and verifying hashes of content on the regular basis, just for the heck of it and after unintended power loss.

    During same years I replaced more than a dozen failed disks on RAID-6 storage and the convenience never stopped to amaze me.  I did have similar hardware with mirror setup - and several times these arrays went offline, because it only takes a timeout for the controller to mark the device bad, and with one device left in mirror - it may happen. (I have to notice that I take "inexpensive" - the "I" in RAID - very seriously, that may have been part of the reason why mirror was not very successful, but on the other hand same or similar disks were used on occasion in RAID-6 too with better success - I never had to take RAID-6 storage offline because of the failed disk)

    Let me ask you - all the bad things that you know about RAID-6 (and I would not use RAID-5 myself) - how many times you saw them actually manifested in your personal experience?

    Did you try to reproduce them in the test environment with modern RAID controllers? If so - were you successful even once ?

    Even for this discussion - in theory SVM mirror can have issue with "write on write" - but, practically, it was NEVER encountered. Furthermore there is an option to synchronize I/O and make SVM safe - right, you guessed it, at the expense of I/O performance.

    One can clearly err on the side of caution, and in the process make the thing practically unusable - like IBM shipping all systems having storage with write cache off.

    Clearly all are welcome to study your paper and other materials about storage technologies, but one has to understand that sometimes attempt to avoid all risks either enormously increases cost + complexity of the solution, or just shifts the risk to the another component / area.






    ------------------------------
    Vladimir Kolobrodov
    ------------------------------



  • 5.  RE: RAID6?

    Posted 29 days ago
    I have seen RAID 6 fail more than once (RAID 5 even more than that) in a way if the system was configured as RAD10 ( properly implemented) we wouldn't have had an outage.  Days of downtime to successfully rebuild and recovery while revenue was lost or deferred. In one case we moved to RAID 10 to limit the future risk.  

    There are so many things that play into solution building. Just like there can be a large difference in performance between SAN and Direct attached storage.  Each has a place based on the solution which is trying to be achieved. As long as the solution designer or implementer is eyes wide open and honest about the risk/reward nothing should be off the table.

    ------------------------------
    ~~~~~~~~~
    Eric Rowell
    ------------------------------



  • 6.  RE: RAID6?

    Posted 30 days ago
    David:

    I have not given up researching. RAID6 is only marginally better than RAID5 as far as data safety and is even SLOOOWWWERRR for write performance than RAID5 since two checksums have to be calculated and written to two different drives. Stick with RAID10.

    That said, you should probably not be using SVM for mirroring RAW chunks or any chunk being used with RAW or DIRECT_IO due to the write-on-write issue. There I agree with HCL tech support. SVM is software mirroring which is the slowest and least reliable. It would be best to use a better LVM that does mirroring properly in parallel or, as suggested, use a good SAN that implements mirroring properly in firmware, again with parallel writes.

    If you MUST use SVM, I would suggest using SVM to create two stripes then use Informix mirroring to mirror them effectively creating a RAID01 array which is as fast and as reliable and nearly as safe as a proper RAID10, just slower to recover from since the entire stripe has to be recovered.

    Art

    Art S. Kagel, President and Principal Consultant
    ASK Database Management


    Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.








  • 7.  RE: RAID6?

    Posted 29 days ago
    Edited by David Grove 29 days ago
    Part way through the tech support case, I decided to disregard SVM, and use the built-in Informix mirroring.  Just for my own edification, I wanted to follow-up with the tech support case to see if there was any firm conclusion available concerning the safety of SVM.  It seemed to me, from information presented from tech support, that Informix didn't do the kind of write-on-write that was most likely what Oracle had in mind in their caution in the 'man metainit' pages.  But, I couldn't be sure because I didn't know for sure what they had in mind.

    In any event, I decided that Informix mirroring could do enough of what we want, and with guaranteed safety, so that's the way we're going.

    I really do appreciate everyone's thoughts.

    At the end of the day, though, as long as there are suitable alternatives ("suitable" meaning effective, safe, and economic), I think we'll continue to pass on RAID5 & RAID6, etc.

    In this present case, we find Informix mirroring to be suitable.

    With full respect to all,

    DG

    P.S.  Vladimir, I meant no disrespect to you or Informix tech support.  (Note that I acknowledged in my initial post that your first recommendation was to use SAN.)

    P.P.S.  I guess I am sort of a pessimist, at heart.  I always feel that the old principle, "If something can go wrong, it will go wrong," is lurking behind me.  So, even if something is unlikely, if there is a way to avoid the possibility (rather than rely on the unlikelihood of an occurrence), I'll choose that.  I think there is sort of a saying among our friends, the physicists, to the effect that (unlikely) phenomena which are not prohibited are guaranteed, eventually, to occur.

    ------------------------------
    David Grove
    ------------------------------



  • 8.  RE: RAID6?

    Posted 29 days ago
    Hi David, Vladi, Art,
    nice discussion but old fashion. I suggest a "waitless i/o/" by using NVMe/Flash storage (local or not). The throughput will depend from RAM and CPU frequence in this case, only. My new test with 14.10 EE, ARM64, Ubuntu on NVIDIA Jeston Xavier NX => 400.000 inserts/second, 50 seconds for a parallel index build on a 100 Mio records table and 650.000 scanread/second. That's the future on a <1.000 $ system :-)

    Best regards,
    Henri

    ------------------------------
    Henri Cujass
    leolo IT, CTO
    Germany
    ------------------------------



  • 9.  RE: RAID6?

    Posted 29 days ago
    Hi, Henri,

    you are absolutely right about new hardware and I/O tech. The problem however remains. If a single (local) device is being used for storage, by Informix or not, it will fail eventually. In some cases it will fail after whole system or storage has been upgraded, however it's very likely to fail when the system is still in operation if the system remains in use for several years. 

    For longer use scenarios there is a need to build some degree of redundancy into the storage solution. With the external storage (SAN), be it based on flash, NVMe, hybrid, of even just disks, such redundancy usually comes with the device, the extra thing to do would be to use multipathing - which would remove single point of failure on the adapter/port level.

    If, for various reasons, external storage solution is not desired, it does make sense to create storage redundancy within local system. And here David is right to use SVM or Informix mirroring for the piece of mind.

    The argument, I guess, emerged from mentioning possibility of using higher RAID levels (5,6), which had (in the past!) quite bad reputation in terms of performance and most importantly the condition known as the "write hole" - which may cause data corruption if power loss happens during write.

    If the decision is made to use redundant storage solution local to the system, next question will be - which level of RAID is more appropriate?

    As usual - there is no straight answer to this question, my comment was only to highlight the fact that stated deficiencies of the higher RAID may not apply for the current hardware, not to say that RAID-6 is better than mirror. (Each solution is "better" in it's own way)

     To illustrate - let's go through several conceptual examples, which use locally attached storage only.

    If you run on SPARCstation 10 or 20 - pretty much your ONLY possible solution will be mirror or SVM.

    Interestingly, if you run on system with the modern motherboard supporting only 2 NVMe drives - the mirror would also be a reasonable choice.

    Note that is both these cases it's not possible to replace failed mirror device without shutting down the system (in SPARCstation it may be possible only if the failed device is external)

    However, if, for example, one has 2U chassis with 8 to 24 disk bays, let's say DELL R740, with the high-end storage controller and more than 6 disks to be used for storage, and these disks are SSD or better, it can be demonstrated that pretty much all the bad things said about RAID-6 may be true but are not practically applicable.

    In fact using separate SSDs with Informix (no redundancy on the device level!) versus volumes carved from RAID-6 (double parity!) over same SSDs demonstrated better performance for the latter. As for the write hole - deployments with UPS, current design of the system's power supplies, and the controller with flash cache and supercapacitor most likely make it a non-issue. Though I cannot say that I had direct experience in this matter, as the occasions when busy systems went down on me due to power loss were extremely rare.

    That's all I was meaning to convey when mentioning RAID-6 as a possible solution, since the old-school arguments for NEVER considering these RAID levels may not be fully applicable in 21 century.

    ------------------------------
    Vladimir Kolobrodov
    ------------------------------



  • 10.  RE: RAID6?

    Posted 25 days ago
    "For longer use scenarios there is a need to build some degree of redundancy into the storage solution."

    Or the other option is to have another compute node and do HDR/RSS!


    ------------------------------
    David Williams
    ------------------------------



  • 11.  RE: RAID6?

    Posted 25 days ago
    Yes, most definitely.

    ------------------------------
    Vladimir Kolobrodov
    ------------------------------



  • 12.  RE: RAID6?

    Posted 29 days ago
    Thank you, Henri.

    I don't think I mentioned it in my posts... but... just as a matter of information, the drives we are using for database storage are SSDs.  (Still retain spinning rust for the O/S, home directories, etc.)

    DG

    ------------------------------
    David Grove
    ------------------------------



  • 13.  RE: RAID6?

    Posted 29 days ago
    The physicist "quote" would be from the experimentalist side of the physics family (the guys that smash things together and try to break them).

    Not that it matters much any more, but, I am still not clear on the potential write-on-write danger with Solaris SVM.  Informix tech support described the situation of a new write occuring before the return is received from a prior write, and then assured us explicitly that Informix does not do that, and never, ever would.  So, I thought, "Ahah! All is good; no problem" (because Informix is the only software that uses the drives in question).

    But, apparently, there is still some danger, in principle.  If there isn't any danger from Informix, then, what could it be?

    In any event, we are no longer planning on using SVM, and use only RAID1 or RAID10 for database storage.

    BTW, are there any LVMs that are considered safe (immune to write-on-write errors)?  Or is hardware RAID (1, 10, or 01) the only way to completely avoid the risk?

    Thank you.

    DG

    ------------------------------
    David Grove
    ------------------------------