Authors: @Ted Hoover, Madhu Thorat @Madhu Punjabi
The convergence of object access to storage and high-performance computing is entering a new phase, driven by the growing demands of AI, GPU-accelerated workloads, and data-intensive analytics. While the S3 API has become the standard interface for modern applications, its reliance on TCP/IP networking has limited its ability to scale efficiently for performance-critical environments.
Starting with IBM Storage Scale 6.0.1, support for S3 over RDMA (Remote Direct Memory Access) is introduced. Leveraging RDMA helps to bypass the kernel and enables direct memory-to-memory data transfers, thus dramatically reducing latency and CPU overhead. The integration of RDMA into S3 represents more than just a performance optimization, it's a fundamental rethinking of how object-based access to storage can serve modern workloads. As AI models grow larger and datasets expand, the ability to move data efficiently becomes as critical as the storage capacity itself.
IBM Storage Scale S3 with RDMA has been developed in collaboration with NVIDIA and leveraging NVIDIA cuObject libraries. This innovation brings object access performance closer than ever to native GPFS file system speeds, without requiring any changes to existing S3 applications. The result is powerful S3 data access support with RDMA, that combines the simplicity of S3 with the performance characteristics required for next-generation AI, HPC, and large-scale data workloads.
With S3 over RDMA, IBM Storage Scale enables:
- Ultra-low latency through kernel-bypass data paths
- Significantly reduced CPU overhead, freeing compute resources
- High-throughput object access to GPFS file systems, approaching file-system performance
- Seamless adoption, with full S3 API compatibility
The combination of IBM Storage Scale's proven reliability and scalability with RDMA's raw performance creates a compelling platform for the most demanding workloads.
Real-world Use Cases
This technology has the potential to revolutionize several areas including:
· AI Training: Faster data loading means reduced training time and better GPU utilization
· High-Performance Analytics: Complex queries over large datasets complete faster with reduced CPU overhead
· Scientific Computing: Simulations and analyses that generate massive datasets benefit from efficient I/O
· Media Processing: Video rendering and processing workflows see dramatic speedups
IBM Storage Scale: One Platform - File and Object access
IBM Storage Scale has long been trusted as a foundation for large-scale, high-performance data infrastructures. It provides a Global Data Platform which allows access to same instance of data through both S3 and file-based access protocols like POSIX, NFS, SMB etc. Through its CES S3 capability, Storage Scale delivers enterprise-grade S3 access to massive datasets, seamlessly integrating object and file-based access within the same storage platform.
How S3 over RDMA Works
S3 over RDMA cleanly separates control and data paths. Authentication, metadata, and control flows remain on HTTP, while bulk object payloads move over RDMA.
The client allocates CPU or GPU memory, registers it for RDMA, and shares a lightweight buffer descriptor with the S3 server. The CES S3 server then transfers data directly into client memory using RDMA, while preserving full S3 compatibility.
The solution is designed to support both traditional CPU-based workloads and GPU-accelerated environments.
Performance Highlights
Testing in IBM Labs demonstrates that S3 over RDMA on InfiniBand, when used with IBM Storage Scale System 6000, can achieve:
· Throughput within ~90% of native POSIX read/write bandwidth
· ~ 300 GB/s object throughput in GPU‑accelerated configurations for S3 GETs
· ~ 155 GB/s object throughput in GPU‑accelerated configurations for S3 PUTs
· Drastically reduced CPU usage and improved efficiency for both clients and CES S3 nodes
· This demonstrated that S3 can now operate at performance levels previously associated only with file system access.
The reduced CPU usage is especially significant for AI workloads, where every CPU cycle saved can be redirected to model training or inference.
References
Storage Scale 6.0.1 Documentation