The Evolution of Block Storage in Ceph

If you're building scalable infrastructure—from cloud-native apps to virtualization or AI pipelines—ceph has long offered a rock-solid foundation. At its core sits RADOS Block Device (RBD)—a tried-and-true, reliable way to carve out block storage from Ceph’s RADOS object store. But as performance demands escalate, particularly for workloads like VMware virtualization, or non-Linux native client access, IBM Storage Ceph inserts NVMe-over-Fabrics (NVMeoF)—specifically NVMe/TCP—into the mix to blur the line between local and distributed storage performance.

1. RBD at a Glance: Solid, Flexible, and Feature-Rich

• Architecture: RBD sits atop RADOS, mapping logical block ranges (LBAs) to objects in the Ceph cluster using librbd and kRBD. Both clients are available today in almost any modern Linux-based operating system.
• Features: Enables snapshots, cloning, striping, object maps, fast-diffs, journaling, deep flattening, and erasure-coded pools. It supports a max virtual device size of 2 EiB, with customizable object-size, stripe-unit/count, and feature sets.
• Access Methods:
• Kernel RBD (kRBD): Kernel compiled Linux driver, works with Linux kernels (≥ 2.6.37) for block device mounts.
• librbd: User-space Linux driver, similar client-side to kRBD, but not compiled into the Linux kernel.
• CSI Drivers: For Kubernetes/OpenShift via IBM Fusion Data Foundation.
• Virtualization: Integrated into OpenStack environments; most KVM-based hypervisors support CephRBD and/or CephFS for VM storage.
• When to Use: Ideal for established, stable block workloads that benefit from rich Ceph-native features—think snapshots, clones, and direct integrated access from within most Linux OSes. Great in container orchestration or opensource virtualization contexts.

2. NVMe-oF (NVMe/TCP) and Ceph: Performance Meets Distributed Storage

• How It Works: IBM Storage Ceph 7.1 Introduced an NVMe-oF gateway to expose RBD images as NVMe namespaces over TCP/IP networks—no need for direct Ceph client libraries. This allows the already stellar core Ceph block storage stack to be leveraged over the industry-standard NVMe/TCP protocol.
• Advantages:
  • Low latency, high throughput: Near-local NVMe performance over standard Ethernet networks.
  • Broad compatibility: Presents volumes as standard NVMe namespaces —simple for RHEL and VMware initiators – can be formatted using any linux-based filesystem (JFS, EXT4, BTRFS, etc.); on VMware as VMFS6 datastores. Also consumable via SmartNICs with NVMe/TCP embedded offload, presenting NVMe namespaces as if they were local to the host.
• Scalability & Resilience:
  • High Availability (HA): Gateway clusters support active/standby configurations, offering I/O resilience and automatic failover.
  • Scale-Out Support: Gateway groups span multiple gateways, subsystems, hosts, and namespaces—suitable for enterprise-scale deployments, and provide more tightly controlled data access.
  • NVMe Discovery: Automatically advertises gateway IPs for initiator discovery, low configuration overhead, minimal Ceph client experience needed.
• Use Cases: Best for latency-sensitive, high-IOPS workloads—critical virtualization environments and high performance structured database workloads. If your app expects enterprise-grade SAN-like block interfaces over TCP, this is your go-to.

3. RBD vs. NVMe-oF: Side-by-Side Comparison

4. Choosing the Right Path

• Use RBD when your Linux-based workloads need rich Ceph-native capabilities—snapshots, cloning, erasure-coding, dynamic resizing—with flexibility and broad orchestration support. This means Linux-native clients, Persistent Volumes in Container environments, and KBM-based hypervisors.
• Opt for NVMe/TCP when your environment benefits from standard NVMe/TCP access—like data-intensive VMware virtualization, high-performance bare metal structured database workloads or where standardizing on a widely adopted OS-agnostic industry protocol is a must.

5. Final Thoughts

Your strategic storage decision—RBD or NVMe/TCP—should align closely with your workload requirements, performance needs, and integration context. What's clear: IBM Storage Ceph empowers you with both robust flexibility and cutting-edge performance.

Want to dig deeper? Discover guides, benchmarks, deployment strategies, and real-world case studies in the comprehensive IBM Redbooks publication "Exploring IBM Storage Ceph Block Storage" (SG24-8591), covering Ceph Version 8.1 (IBM Redbooks).

Ready to Elevate Your Storage Strategy?

Explore the full breadth of IBM Storage Ceph capabilities—RBD, NVMe/TCP, deployment best practices, scalability, and performance—over at https://www.ibm.com/products/storage-ceph Get hands-on with modern block storage for your most demanding workloads today.

Going to TechXchange 2025?

Don’t miss our exciting IBM Storage Ceph Sessions and Labs such as:

Ceph block performance with VMware NVMe/TCP [2892]

IBM Storage Ceph NVMe over TCP: VMware Integration and Data Migration [2621]

Using Ansible to manage IBM Storage Ceph [1375]