Ceph Object Storage access with Elbencho

Elbencho is a powerful tool for benchmarking distributed storage systems. 

It supports various protocols, including S3, and is well-suited for testing object storage performance on platforms like Ceph RGW. It measures IOPS, throughput, and latency for S3-compatible storage.

Here's how to set it up and run some basic benchmarks.

Prerequisites for RHEL-8 or higher

 Ensure the following dependencies are installed on your system:

• Boost Libraries: sudo yum install boost-devel

• Build Tools: gcc-c++, make, cmake

• Additional Libraries: libaio-devel, openssl-devel, zlib-devel, python3-devel, libcurl-devel

Quick Setup to test S3 Ceph RGW. 

Download Elbencho Static Binary

wget https://github.com/breuner/elbencho/releases/download/v3.0-25/elbencho-static-x86_64.tar.gz

tar -xf elbencho-static-x86_64.tar.gz

Verify Installation

Confirm the binary works:

./elbencho --help

./elbencho --help-s3

Running Basic S3 CRUD Tests

Test 1: Directory Creation in an S3 Bucket

./elbencho --s3endpoints http://<S3_ENDPOINT>:<PORT> \ --s3key <ACCESS_KEY> \ --s3secret <SECRET_KEY> \ -d mybucket1

Example output 

• What does it do?

• This command creates a "directory" in the specified bucket (mybucket1).

• Key Flags:

• --s3endpoints: Specifies the S3 endpoint URL.

• --s3key & --s3secret: Authentication details (access and secret keys).

• -d mybucket1: Designates the bucket where the directory is being created.

Test 2: Writing Objects to an S3 Bucket

Test 2 threads, each creating 3 directories with 4, 10MB objects per directory

./elbencho --s3endpoints http://<S3_ENDPOINT>:<PORT> \ --s3key <ACCESS_KEY> \ --s3secret <SECRET_KEY> \ -w -t 2 -n 3 -N 4 -s 10m -b 5m mybucket1

Example output 

• What it does:

• Writes objects to the specified bucket (mybucket1).

• Objects are divided into parts if they're large (multi-part uploads).

• Key Flags:

• -w: Enables write mode (writing objects to the bucket).

• -t 2: Spawns 2 threads to operate.

• -n 3: Each thread writes 3 objects, resulting in 6 total (2 threads × 3 objects).

• -N 4: Writes are divided into 4 parts (multipart upload).

• -s 10m: Sets the object size to 10 MB.

• -b 5m: Sets the block size (chunk size) to 5 MB.

Test 3: Read the Objects

./elbencho --s3endpoints http://<S3_ENDPOINT>:<PORT> \ --s3key <ACCESS_KEY> \ --s3secret <SECRET_KEY> \  -r -t 2 -n 3 -N 4 -s 10m -b 5m mybucket1

Example Output

• Operation: Reading files from mybucket1.

• Elapsed time: Duration of the read operations (1.250s to 1.452s).

• Files/s: Number of files read per second (16 files/s).

• IOPS: Read operations per second (33-34 IOPS).

• Throughput (MiB/s): Data throughput during the read operation (165-171 MiB/s).

• Total MiB: Total data read (215-240 MiB).

• Files total: Total number of files read (21-24 files).

The read performance is significantly better than the write, as shown by higher throughput, IOPS, and file processing rates.

Test 4: Deleting Files and Directories

Clean up the bucket by deleting files and directories:

./elbencho --s3endpoints http://<S3_ENDPOINT>:<PORT> \ --s3key <ACCESS_KEY> \ --s3secret <SECRET_KEY> \ -D -F -t 2 -n 3 -N 4 mybucket1

Example output 

• What it does:

• Deletes the objects and directories created in the bucket (mybucket1).

• Key Flags:

• -D: Deletes objects.

• -F: Deletes directories after removing files.

• -t, -n, and -N: Behave the same as in the write test, controlling threads, object count, and multipart settings.

Conclusion

Elbencho is best suited for POSIX/S3 performance analysis, detailed directory operations, and object ingestion testing.