Originally posted by: TonyPearson

Wrapping up my week on All-Flash arrays, I thought I would cover some of the Enterprise Reliability features of the IBM FlashSystem.

On Monday, [IBM FlashSystem versus EMC XtremeIO all-Flash Arrays], I discussed some of the features of the IBM FlashSystem that differentiate it from EMC's ExtremeIO and other all-Flash arrays. On Tuesday, [IBM 2013 Storage Announcements for November 19] included discussion of the all-Flash model of the IBM System Storage DS8870 disk system.

Just as light bulbs burn out eventually after repeatedly being turned on and off, Flash does not last forever either.

A set of transistors can represent a single bit of informaiton (Single-level cell, or SLC for short), or multiple bits (Multi-level Cell, MLC). MLC typically refers to two bits, with a new "Triple-level cell" or TLC technology, able to store three bits per set of transistors.

SLC is faster and can endure more "Program-erase" write cycles, but MLC is less expensive to manufacture and therefore used in most consumer products, like digital cameras, smart phones, music players and USB memory sticks. To learn more on this, see this 6-page IBM whitepaper on [Comparison of NAND Flash Technologies Used in Solid-State Storage].

In between, "Enterprise MLC" (or eMLC for short) refers specifically to a different grade of chips IBM gets from the flash manufacturer. eMLC chips use a similar MLC bit arrangement, but are typically selected from higher bins, and most importantly have much longer program-erase cycle times which yield greater chip endurance, at the expense of long data retention when power is off (but seriously, when is anything off for very long in a data center?)

As a result, eMLC has 10x the endurance of regalar MLC, approaching parity with SLC at half the cost!

In the IBM FlashSystem, DRAM cache is used to buffer the writes first, then written out to the Flash. This helps to further improve the endurance.

For enterprise reliability, each Flash chip on the IBM FlashSystem has Error Correcting Codes (ECC), and then each set of 10 chips is placed in a 9+P RAID-5 configuration.

The chips are sub-divided into 16 planes. In the event a cell fails, the data for that plane can be reconstructed from parity, and written to spare space on the other planes of that same chip set. That plane is then reformated as an 8+P RAID-5, bypassing the failed plane.

In this manner, a cell failure only results in losing a small portion of one chip. If the same plane fails another failure on another chip, it will drop down to 7+P, 6+P, 5+P, and finally 4+P. This is known as "Variable Stripe RAID" or VSR for short.

IBM FlashSystem can survive over 1,000 such cell failures without an outage. By comparison, a single cell failure on an SSD often marks the entire drive as a failure.

But wait, there's more. Why stop at just RAID-5 across 10 chips. The chips are organized into modules, and IBM FlashSystem can perform RAID-5 across modules, in a 10+P+S RAID-5 configuration. This is referred to as "Two dimensional RAID" or 2D-RAID for short.

Even if you lost an entire module, the system will automatically rebuild on the spare module, and you can replace the bad one non-disruptively.

Many use cases for all-Flash arrays do not require such high levels of Enterprise reliability. Several of the all-Flash competitors have adopted a "design-for-failure" approach common among Cloud Service Providers like Amazon Web Services.

The idea is to assume that the data stored on them is just a copy from some other storage media. In the event of a Flash failure, it can easily be restored from a mirrored copy or backup.

For the IBM FlashSystem, The newer 800 series are based on eMLC, ideal for the majority of business applications, databases and virtual machine images placed on all-Flash arrays. The older 700 series are based on more expensive SLC, designed specifically for sustained write-intensive workloads.

Within each series, the "tens" models (710, 810) offer RAID-0 striping across ECC and VSR protected modules. For higher levels of availability, the "twenties" models (720, 820) offer ECC, VSR and 2D-RAID protection.

technorati tags: IBM, FlashSystem, SSD, Flash, endurance, high availability, enterprise reliability, SLC, MLC, eMLC, ECC, VSR, 2D-RAID, EMC, XtremeIO, DS8870