Author: Thomas Stober - TSTOBER@de.ibm.com
IBM® Fusion Data Foundation in a nutshell
IBM Fusion Data Foundation is an underlying storage infrastructure of IBM Fusion. It provides physical storage to containerized workload using abstracted, easy-to-use persistence services for stateful containerized applications.
There are 3 specific attributes of this software-defined storage infrastructure:
- Consistent user experience: By using well-known open-source components, you can take advantage of abundant skills in the market and easily install IBM Fusion Data Foundation using operators from the Red Hat® OpenShift® Container Platform web user interface. Being tightly integrated into Red Hat OpenShift, you can unlock the Red Hat ecosystem for your workload. A consistent operation across hybrid cloud and the ability to manage and monitor from a single dashboard are a few keys to efficiency.
- Dynamic scale: IBM Fusion Data Foundation unlocks your heterogenous infrastructure and lets it grow and expand as needed independently of the application development. IBM Fusion Data Foundation is built on mature Red Hat Ceph® technology that is proven to scale to tens of petabytes in production. This allows IBM Fusion Data Foundation to scale linearly as workloads increase.
- Simplified access: IBM Fusion Data Foundation allows clients to deploy and consume storage wherever it is needed in a uniform fashion. You can share your data with applications across a hybrid cloud environment as well as on premises. All major storage protocols for container workloads are supported, which makes IBM Fusion Data Foundation suitable for any kind of application. IBM Fusion Data Foundation is a complete integrated solution, which covers the entire data lifecycle.
IBM Fusion Data Foundation consists of the following open-source core components:
- Ceph is a distributed storage system, which takes care of managing disks and logical storage units.
- Rook is an orchestrator, which deals with configuration and life cycle of the persistent volumes. It makes dealing with Ceph significantly easier, as it applies a set of opinionated settings.
- NooBaa is the multi-cloud gateway, which provides S3 endpoints, when storing objects in external cloud offerings.
IBM Fusion Data Foundation supports a variety of storage classes, which gives developers flexibility when implementing workload. Storage classes allow administrators to describe the characteristics of the storage they offer. Different classes might map to quality-of-service levels, backup policies, or arbitrary policies determined by the cluster administrators.
For access of data, characteristics can be distinguished depending on how they are shared or non-shared across Red Hat OpenShift nodes.
- ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node.
- ReadOnlyMany (ROX): The volume can be mounted read-only by many nodes.
- ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes.
File, block, and object storage classes are possible to be consumed by stateful containerized applications.
- Block storage is mainly suitable for databases and systems of record.
- File storage (shared and distributed file system) is designed for multiple threads and multi-tenancy, and suitable for messaging, data aggregation, machine learning, and deep learning.
- Object storage is suitable for images, non-binary files, documents, snapshots, or backups. In IBM Fusion Data Foundation either Ceph or the Nobaao Multi Cloud Gateway can be used for Object storage. Both options expose the well-known S3 protocol for access.
How IBM Fusion Data Foundation works
Let us now have a look, how IBM Fusion Data Foundation is working under the hoods:
IBM Fusion Data Foundation runs inside a Red Hat OpenShift cluster like any other Kubernetes workload and can be scaled up and down as needed. Fusion Data Foundation pods are referred to as storage nodes and can be deployed either on compute or infrastructure nodes. You will need 3 storage nodes for a complete IBM Fusion Data Foundation installation.
Containerized application workload runs on compute nodes. If the application desires to store its state and data, it will create a persistent volume claim (PVC). Kubernetes links the application container with a corresponding container-native storage pod. As part of that linking, a PVC will be bound to Persistent Volumes (PV) of the desired type of storage class. A PV is the logical representation of a storage unit. It is not real physical storage yet.
IBM Fusion Data Foundation will now orchestrate the magic of mapping real physical storage to the used PVs. It will also ensure that there will always be 3 copies of the data to ensure data consistency. An entity of the managed storage is referred to as Object Storage Daemon (OSD). IBM Fusion Data Foundation takes advantage of the local storage operator of Red Hat OpenShift to interact with the physical storage.
Furthermore, IBM Fusion Data Foundation provides additional capabilities for seamless operation, like monitoring of the storage.
Adding resilience and disaster recovery
In failure scenarios, storage nodes might become unavailable to the Red Hat OpenShift cluster. This could be caused by unavailability of the LPAR or physical hardware, by the failure of a storage node or physical storage, or by network failures interrupting the interaction between the nodes within the cluster.
Red Hat OpenShift and IBM Fusion Data Foundation provide data protection and resiliency capabilities to mitigate such failures.
The high availability (HA) capabilities of a single IBM Fusion Data Foundation cluster can mask unplanned outages from end-users. Failover can be realized by stretching a single cluster across multiple availability zones if a reliable inter-node communication with low latency can be guaranteed and at least 2 storage node instances are operational.
If the customer requirement is to deliver a non-disruptive service to end users 7 days a week, 24 hours a day, without any planned or unplanned outages even in the case of a disaster, which would eliminate an entire data center, additional measures need to be put in place:
A full disaster recovery solution is typically realized with 2 or more data center sites, with an own Red Hat OpenShift and a dedicated IBM Fusion Data Foundation deployment on each site. Depending on the geographical distance between the 2 data centers, such a “golden topology” can be implemented either as Regional Disaster Recovery (RDR) solution, or as Metro Disaster Recovery (MDR) solution.
Regional Disaster Recovery assumes one active site, which runs the application workload. The other site is on warm standby and will only be activated in case of the first site is failing. Both sites will have the same persistent data, which is replicated between the two. The replication is asynchronous and can be performed even across large geographical distances, which allows the 2 data centers to be located far apart from each other.
Metro Disaster Recovery assumes that the two data centres are in proximity of each other, and low network latency allow for a reliable synchronous replication. In this case both data centres are active at the same time.
Figure 1: Options for high availability and disaster recovery
Each of these 3 distinct topologies have their own characteristics and specific use cases.
The simplest option depicted in figure 1 is the cross rack high availability topology on the left side: in this option, you can deploy a single stretched Red Hat OpenShift and Fusion Data Foundation cluster across multiple LPARs and multiple CECs. This topology is easy to set up and easy to manage.
As mentioned above, you can define availability zones and guarantee a low latency for the network connectivity between the involved nodes of the cluster. In case of a single failing availability zone, the system will continue to function properly. The usual mechanism of Kubernetes orchestration will be able to deal with that situation immediately and automatically. The Recovery Point Objective (RPO) is zero, there is no data loss. But the outage of multiple zones or even an entire data center (in case of a disaster) cannot be recovered with this approach.
The second option is the Regional Disaster Recovery approach with the asynchronous replication between the two independent clusters and data centers. In this setup, both clusters need to be deployed and must always run to make the replication work properly. Nevertheless, only one cluster is performing active duty, while the other one is in stand-by mode.
An important additional component is added here: Red Hat Advanced Cluster Management for Kubernetes (ACM) will perform the orchestration between the two independent Red Hat OpenShift clusters and coordinate the failover from one cluster to the other.
As the data replication under the hoods is asynchronous the Recovery Point Objective (RPO) is in the range of minutes. On the other hand, the network latency between the two clusters doesn’t matter at all, which allow the data centers to be located far away of each other, e.g. in two different regions.
Let us now look at the third option, the Metro Disaster Recovery: again, ACM is the component to orchestrate the disaster recovery setup. Again, there are two independent Red Hat OpenShift clusters deployed. But this time a single storage deployment storage is stretched across the 2 datacenters. That storage is provided by IBM Storage Ceph on a x86 machine. Inside that stretched cluster there is a synchronous replication happening between the nodes in the 2 data centers. This gives us the Recovery Point Objective RPO of zero!
Due to the synchronous replication, there is a need for low latency network communication between the storage nodes of Ceph.
An alternate option to achieve disaster recovery is to mirror the underlying physical storage hardware (agnostic to the running applications). But the downside of this approach is that the mirroring between 2 data centers will always be a full cloning of an entire cluster deployment, rather than the smart application scoped replications provided by the other options described above.
Some thoughts on container-native thinking
When dealing with persistence of container-based applications, it’s worth spending some thoughts how to take advantage of containers and Kubernetes. This is especially important when resilience and disaster recovery are essential considerations of your solution.
In general, containers are small, lightweight bundles, which execute one or more applications. They include the minimal amount of operating system and system resources, but all required dependencies like runtime environment, libraries, and infrastructure services.
Containers are stateless without any persistent data. This limitation is imposed to keep a light footprint. As containers don’t have to bother with persistence they can be easily distributed and scaled across virtual and physical server as well as data centers to meet the customer’s needs in real time. The risk of data conflicts is avoided. The application will always start in the same initial state, which has been defined by the application developer in the first place. It is essential for resilience purposes that the initial state of the entire production environment including all application workload can be automatically restored and deployed anywhere, at any time, and as often as needed – even after a crash or a disaster.
The best place to safeguard defining information of the initial state of an application like code or meta data configuration is a code repo like Github. It will only be changed as part of a well-defined CI/CD process of agile software development.
But what about the stored data representing the always changing state of applications? Once your applications become stateful and require persistent storage, then its prime time for software-defined storage: in the world of Kubernetes, the application data is stored in persistent volumes. Its responsibility of a software-defined storage infrastructure like Fusion Data Foundation to make those persistent volumes available to the corresponding applications. This implies to share and replicate the data across all nodes or systems, on which a stateful application is running in parallel (e.g. as part of the regional disaster recovery solution). This also includes the need for safeguarding persistent volumes in backup archives.
When creating stateful containerized applications, it is important to clearly separate information describing the initial application state from the corresponding persistent application data.
You can restore an entire application namespace based on the application state (defined as code and meta data which is stored in a code repository and part of a reproducible CI/CD process) and the persistent application data (saved in persistent volumes managed as backup archives by IBM Fusion Data Foundation).
Support for NFS
In many customer applications, the existing storage infrastructure is based on NFS. To add the benefit of software-defined storage without modifying the application, IBM Fusion offers an NFS access layer on top of IBM Fusion Data Foundation. This adds resiliency, performance, and backup/restore to the solution with major rework of the legacy applications.
Support for Encryption
Fusion Data Foundation supports two levels of encryption:
- an entire logical Object Storage Daemon (OSD) can be encrypted,
- or individual persistent volumes can be encrypted (with individual keys each).
The keys used for the encryption can be stored either
Internal key storage uses the etcd database of the Red Hat OpenShift cluster for safeguarding encryption keys.
The other option is to store keys externally in a dedicated key management system, which is much more advisable and secure. Currently the recommended and supported key management system of choice is a HashiCorp vault.
When encrypting an entire OSD encryption, both options for key storage can be used locally in etcd, as well as externally in HashiCorp vault. The encryption is done by IBM Fusion Data Foundation using LUKS mechanism.
For encrypting induvial persistent volumes, only the external key management option is supported. The encryption mechanism used here is also LUKS. It is possible to encrypt individual persistent volumes with an individual unique key. This gives you the flexibility to scope application data to specific tenants only. Each tenant will only have access to the keys for the persistent volumes he is authorized to use.
Replica-1: reducing the data replication between storage nodes
IBM Fusion Data Foundation (FDF) allows to remove the software-based resilience capabilities and rely on underlying hardware only. In this “replica-1” option, you can configure OSDs to disable the usual replication, which keeps the data in 3 redundant copies on 3 storage nodes (see figure 2). Therefore, data is stored only once, which reduces the demand for disk size and speeds up I/O access.
This option is interesting for IBM Z and IBM LinuxONE clients, who have highly reliable physical disks systems in place and want to avoid the double redundancy added by Fusion Data Foundation.
Figure 2: Configuring a replica-1 OSD for IBM Fusion Data Foundation
Information about software-defined storage on IBM Z and IBM LinuxONE can be found at: Software-defined storage on IBM Z and IBM LinuxONE