1. Introduction
As many you might ask yourself what are the major differences between RedHat OpenShift Data Foundation (ODF) Internal vs External mode. Through this post I will try to provide you with more details about the ODF solution that is part of the Spectrum Fusion offering and provide a summary of the key differences between the two deployment models. This document focuses on Red Hat OpenShift 4.10.x and Red Hat OpenShift Data Foundation (4.10.x) as this is the latest supported version For IBM CloudPaks as I put this document together.
N.B.: This document voluntarily sets aside the Multicloud Gateway deployment and configuration and focuses solely on the persistent storage backend powered by the Ceph technology.
2. Deployment Models
2.1 Strategy
Red Hat ODF leverages the Rook operator to perform the initial deployment of the underlying Ceph cluster but also to perform various configuration task that are part of day-2 operations such as:
- Expand the Ceph cluster
- Apply Kubernetes pod configuration post-deployment
- Redeploy a Ceph OSD
The deployment of a Red Hat ODF will vary depending on the size of the cluster to deploy as well as based on the deployment model selected. It can vary from about 10 minutes to some hours depending on what part of the deployment are being considered (Red Hat OpenShift portion only to Red Hat OpenShift and a fully-fledged Red Hat Ceph Storage cluster acting as an external Red Hat ODF cluster. Red Hat ODF supports the concept of taints and tolerations to further control coexistence of workloads on specific nodes.
The Red Hat ODF operator uses standard Kubernetes resource request and limit specifications for most of the Red Hat ODF pods. The official planning guide from Red Hat provides the following table for Internal Mode Deployment models. The resources outlined are Red Hat OpenShift node resources requirements.
| Deployment Model |
Base Services |
Additional Device Set |
| Internal |
- 30 CPUs (logical)
- 72 GiB memory
- 3 storage devices (1 per node)
|
- 6 CPUs (logical)
- 15 GiB memory
- 3 storage devices (1 per node)
|
| External |
- 4 CPUs (logical)
- 16 GiB memory
|
|
2.2 Internal Converged Deployment Model
The Rook operator deploys the Ceph cluster on the worker nodes selected by the administrator. As such the application workload pods will coexist with the Red Hat ODF pods on the selected worker nodes.
2.3 Internal Infrastructure Deployment Model
The Rook operator deploys the Ceph cluster on the infrastructure nodes selected by the administrator. As such the application workload pods will not coexist with the Red Hat ODF pods on the worker nodes.
2.4 Internal Compact Deployment Model
The Rook operator deploys the Ceph cluster on the control plane. In this deployment model the control-plane nodes carry both a control-plane as well as a worker label. The application workload pods coexist with the Red Hat ODF pods on the nodes and the control-plane specific pods.
2.5 External Deployment Model
The Rook operator deploys the Ceph CSI provisioner and plugin pods. The configuration data collected on the external Red Hat Ceph Storage cluster will be used to provision and attach physical volumes to the pods requesting a Persistent Volume Claims from one of the Red Hat ODF storage classes.
3. External vs Internal Comparison
3.1 Resource Consumption
As each internal cluster requires a minimum allocation of resources through the Kubernetes resource request specifications set by the operator, even a cluster that is almost empty will still require the minimum number of resources. Whether the cluster hosts 1 byte or 3TiB of usable data, the number of resources required to perform the deployment in terms of CPU and RAM is identical.
If using an external mode deployment model, the resources of the entire Red Hat Ceph Storage cluster will be mutualized to serve all the Red Hat OpenShift cluster attached to it with an out of the box Red Hat Ceph Storage configuration.
3.2 Ceph Deployment and Configuration
Red Hat ODF deploys Ceph in a very opinionated way and applies pre-tested configuration settings when running in a Red Hat OpenShift environment. As a result, the deployment of an internal cluster is extremely easy and swift while limiting the tuning that can be performed at the Ceph level.
When using the external deployment model, the Red Hat Ceph Storage cluster must be deployed using the supported deployment tools only. However, as the cluster is externally deployed and managed, any aspect of Ceph can be tuned (e.g. CRUSH configuration). However, once the external cluster is deployed, all subsequent Red Hat OpenShift clusters can consume it with minimal extra configuration.
3.3 Underlying Devices
Red Hat ODF is only supported when backed by underlying flash devices. All devices in the cluster must be identical in size and type. Red Hat Ceph Storage clusters support being deployed with different host types and different storage device types to be better aligned with the various SLAs or use cases present on site.
3.4 Platform Support
Red Hat ODF can be deployed on many platforms not supported by Red Hat Ceph Storage.
Red Hat ODF can specifically be deployed on public cloud environments while Red Hat Ceph Storage cannot be deployed in public cloud environments. As Red Hat Ceph Storage is moving to the IBM portfolio together with the product management and strategy this will likely undergo some changes and a better alignment.
3.5 Administration
Red Hat ODF provides a Red Hat OpenShift integrated solution with an integrated dashboard. Most day-2 operations are being taken care of by the Rook operator.
An external Red Hat Ceph Storage cluster is a separate entity that requires the administration tasks to be carried outside of the Red Hat OpenShift cluster and presents a separate dashboard. All administration tasks require the use of the Ceph specific tooling or UI.
4 Decision Matrix
|
Item
|
Internal
|
External
|
|
Device type supported
|
Flash only
|
Flash or HDD (non SMR)
|
|
OSD individual capacity
|
T-Shirt sizes (.5, 2 and 4 TiB)
|
No limit
|
|
Multiple OSD size
|
No
|
Yes
|
|
Multiple OSD per drive
|
Yes (LSO only)
|
Yes
|
|
OSD external storage
|
Yes through CSI drivers
|
Requires support exception
|
|
OSD metadata segregation
|
Yes (requires CLI deployment)
|
Yes
|
|
Ceph customization
|
Limited (replication factor)
|
Full
|
|
Data placement
|
Non customizable
|
Fully customizable
|
|
Erasure Code support
|
No
|
Yes (S3 only for OCP workloads)
|
|
Deployment tool
|
Rook
|
cephadm
|
|
Network segregation
|
Yes (Multus TP)
|
Yes (Native)
|
|
ODF cluster mutualization
|
No
|
Yes
|
|
Multicloud gateway
|
Yes
|
Yes
|
|
S3-SSE
|
Yes (through MCG)
|
Yes (through RGW)
|
|
Multiple storage class support
|
Yes
|
Only one set per OCP cluster
|
|
Stretched cluster
|
No
|
Yes
|
|
K8S scheduler constraints
|
Yes
|
No
|
|
Public cloud support
|
Yes
|
No
|
|
On the wire encryption
|
Yes (S3 MCG only)
|
Yes
|
|
At rest encryption
|
Yes
|
Yes
|
|
PV Level encryption
|
Yes (RBD only)
|
Yes (RBD only)
|
|
External KMS support
|
Yes
|
Yes
|
#Highlights-home#Highlights