Storage

Introduction

In a world where data must be quickly accessed and protected across multiple Geos, multi-site replication is a critical feature of object storage. Whether running global applications or maintaining a robust disaster recovery plan, replicating objects between different regions is essential for redundancy and business continuity.

IBM Storage Ceph Object storage multi-site replication is asynchronous log-based replication. The nature of Async replication can present challenges validating where your data currently resides or confirming that it has fully replicated to all remote zones; this is not acceptable for certain applications/use cases that require close to strong consistency on write, and all the objects to be replicated and available in all sites before they are made accessible to the application/user.

As a side note, Complete consistency on write can be provided using an IBM Storage Ceph Stretched Cluster solution that provides synchronous replication, but this has its limitations for a geo-dispersed solution as latency is a key factor for stretched clusters, so if you need geo-replication you are looking at multisite async replication for object storage as your solution.

The Challenge

Application owners often need to know if an object is already available in the destination zone before triggering subsequent actions (e.g., analytics jobs, further data processing, or regulatory archiving). Storage Operations teams may want clear insight into how long replication takes, enabling them to alert and diagnose slow or faulty network links. Automation and Data Pipelines might require a programmatic way to track replication status before proceeding with the next step in a workflow.

IBM Storage Ceph 8.0: Introduces Replication Headers

To address this visibility gap, IBM Storage Ceph Object Storage 8.0 introduces new HTTP response headers that expose exactly where each object is in the replication process:

1. x-amz-replication-status
   A quick way to determine if the object is pending replication, in progress, or already fully replicated (status might read PENDING, COMPLETED, or REPLICA, etc., depending on configuration).

2. x-rgw-replicated-from
   Shows the source zone from which the object was initially ingested.

3. x-rgw-replicated-at
   Provides a timestamp indicating when the object was successfully replicated. By comparing this to the object’s Last-Modified header, you get an instant measure of replication latency—valuable for real-time monitoring or performance tuning.

With these headers, you have a programmatic and deterministic way to know whether the data has propagated to the target zone. It’s vital to notice that the primary use case for these new HTTP replication headers is to query the status of objects during ingest to help the application make decisions based on the replication status of the objects, although possible this is not a feature for the Storage Infrastructure team to check the replication status of all objects scanning through billions of objects to check their status, that usage of the headers seems far from ideal.

Example use cases for the HTTP replication headers

Application Workflows

Developers can integrate these headers into their application logic. After uploading an object, the application can “poll” for the x-amz-replication-status to ensure the object is fully available in the destination zone before triggering subsequent steps.

Operations Canary Monitoring

An automated job (or “canary test”) can periodically upload and delete an object, checking how long replication takes. If latency breaches a certain threshold, it can alert the operations team, flagging potential network or configuration issues.

Data Pipelines and Notifications

While polling headers is often the most straightforward approach, you may leverage S3 bucket notifications in IBM Storage Ceph for certain replication-related events. Integrating these with a message broker like Kafka can help orchestrate larger, event-driven workflows. (See the Ceph docs on S3 notifications)

Basic Hands-on Example

I have a multi-site replication setup between two IBM Storage Ceph clusters. They are part of a zonegroup called multizg, and we have bi-directional full-zone replication configured between zone1 and zone2

# radosgw-admin sync info
{
    "sources": [
        {
            "id": "all",
            "source": {
                "zone": "zone2",
                "bucket": "*"
            },
            "dest": {
                "zone": "zone1",
                "bucket": "*"
            },
"dests": [
        {
            "id": "all",
            "source": {
                "zone": "zone1",
                "bucket": "*"
            },
            "dest": {
                "zone": "zone2",
                "bucket": "*"
            },
...

For detailed information about IBM Storage Ceph Object Storage multisite replication, we have a blog series that covers this feature in-depth, from Architecture to setup and fine-tuning; check it out here.

A simple way to view these new replication headers is to use s3cmd with the --debug flag, which prints raw HTTP response headers from the Ceph Object Gateway. By filtering for “rgw-” or “x-amz-” lines, we can easily spot replication-related information. 

Let's check it out. I uploaded an object into zone1:

# s3cmd --host ceph-node-00:8088 put /etc/hosts s3://bucket1/file20
upload: '/etc/hosts' -> 's3://bucket1/file20'  [1 of 1]
 640 of 640   100% in    0s     7.63 KB/s  done

If I check the object's status on the source site where I uploaded the object, it’s in a PENDING state, indicating the object is still replicating. Eventually, once replication is complete, the status will transition to COMPLETED the source zone and REPLICA to the destination zone.

# s3cmd --host ceph-node-00:8088 --debug info s3://bucket1/file20 2>&1 | grep -B 2 'rgw-'
             'x-amz-replication-status': 'PENDING',
             'x-amz-request-id': 'tx00000f2948c72a2d2fb8e-0067a5c961-35964-zone1',
             'x-rgw-object-type': 'Normal'},

Now, let’s check on the destination zone endpoint:

# s3cmd --host ceph-node-05:8088 --debug info s3://bucket1/file20 2>&1 | grep -B 2 'rgw-'
             'x-amz-replication-status': 'REPLICA',
             'x-amz-request-id': 'tx00000a98cf7b6a584b95b-0067a5cac9-29779-zone2',
             'x-rgw-object-type': 'Normal',
             'x-rgw-replicated-at': 'Fri, 07 Feb 2025 08:50:07 GMT',
             'x-rgw-replicated-from': 'b6c9ca95-6683-42a5-9dff-ba209039c61b:bucket1:b6c9ca95-6683-42a5-9dff-ba209039c61b.32035.1'},

Here, the relevant headers tell us:

• x-amz-replication-status: REPLICA
  The object has completed replication and is available in the remote zone(s).

• x-rgw-replicated-at: 'Fri, 07 Feb 2025 08:50:07 GMT’
  Shows the timestamp when replication was finished.

• x-rgw-replicated-from: 8f8c3759-aaaf-4e6d-b346-...:bucket1:...
  Identifies the source zone (and bucket) from which the object was replicated.

Let's check that the status of the object in the source site has moved into COMPLETE state:

# s3cmd --host ceph-node-00:8088 --debug info s3://bucket1/file20 2>&1 | grep x-amz-replication-status
             'x-amz-replication-status': 'COMPLETED',

Example of how to use the headers in the application workflow

This straightforward polling mechanism—via HEAD or info requests—can be automated into application workflows to confirm full replication before taking further actions. Let’s check out a basic example.

Imagine a Content Delivery Network (CDN) scenario where you must replicate files globally to ensure low-latency access for end users across multiple regions/zones. An application in one region uploads media assets (images, videos, or static website content) that must be replicated to other zones before we can make them available to the end users for consumption.

Here is a code snippet with an example of using Python boto3 to upload media content to a site, and pole the replication status of our new uploaded media content by quering the replication status header, once the object has been replicated we print out relevant information, source , destination zone and replication latency.

Application Example output:

Conclusion

The new replication headers in IBM Storage Ceph Object Storage 8.0 mark a significant step forward in giving developers, DevOps teams, and storage administrators more granular control and visibility over multi-site replication. By querying the x-amz-replication-status, x-rgw-replicated-from, and x-rgw-replicated-at headers, applications can confirm that objects have fully synchronized before proceeding with downstream workflows. This simple yet powerful capability can streamline CDN distribution, data analytics pipelines, and other use cases hinge on multi-site consistency.

Check out the official IBM Storage Ceph Object Storage documentation, or contact your IBM representative for more details. Feel free to share your experience with the IBM Storage community if you have feedback or need guidance.