IBM Storage Ceph Object Storage Multisite Replication Series. Part Eight. Archive Zone Data Recovery Example

In part Eight of this IBM Storage Ceph Multisite series, we will keep working with the Archive Zone; using a hands-on example of recovering data from the archive zone, we will cover in detail how the archive zone works.

Introduction

Let's start with a visual representation of the recovery workflow for the archive zone; once covered, we will follow the same steps with a hands-on example.

The following diagram shows the behaviour of the Archive zone when a user PUTs an object into the production zone.

1. With the first PUT of object1 from the user into the production zone, the object replicates into the archive zone as the current version object.
2. When the user issues a second PUT that updates object1, the new modification gets replicated to the archive zone. The new modified object is now the current version and the old object is still available in the archive zone thanks to S3 versioning.
3. If the user issues a third PUT that updates object1, the same will happen as in step2, and we will have three versions of the object available in the archive zone.

If we continue with the same example we have depicted above, let's check how we could recover data from a logical failure.

1. During the application X rollout, A user introduces an operational mistake and deletes object1 in the production zone; the objects don't get deleted from the archive zone.
2. When our production application X tries to access object 1, it can't access the object, the application is down, and panic starts.
3. The APP team RCAs the issue, and they give a quick call to the Storage team specifying the object and what version(Date/time) they want to recover from.
4. The storage team copies object 1's latest version from the Archive zone into the production cluster.

Hands-on Object Recovery from the Archive Zone Workflow

Prepare the client Environment.

We are going to use rclone cli tool. For our testing, first, we are going to create a specific user for our tests, so in our zone1 cluster, we run:

# radosgw-admin user create --uid=archuser --display-name="S3 user to test the archive zone" --access-key=archuser --secret-key=archuser
{
    "user_id": "archuser",
    "display_name": "S3 user to test the archive zone",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "archuser",
            "access_key": "archuser",
            "secret_key": "archuser"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {

We are going to configure the AWS client with this user:

# aws configure
AWS Access Key ID [None]: archuser
AWS Secret Access Key [None]: archuser
Default region name [None]: multizg
Default output format [None]: text

I will also create a couple of aliases to make our life easier.

Alias for zone1 and the archive zone:

# alias s3apiarchive='aws --endpoint=https://object.s3.archive.dan.ceph.blue:443 s3api'
# alias s3apizone1='aws --endpoint=https://object.s3.zone1.dan.ceph.blue:443 s3api'

We will also use rclone, so let's download and install the rclone RPM:

# yum install https://downloads.rclone.org/v1.62.0/rclone-v1.62.0-linux-amd64.rpm -y

Configure the rclone client, with our production zone endpoint and the archive zone endpoint; in this way, with the use of rclone we can recover data from the archive zone if required

Create some test files and get the output of their MD5 sum, so we can review further down the line:

# echo "This is file 1" > /tmp/test-file-1
# echo "This is file 2" > /tmp/test-file-2
# echo "This is file 3" > /tmp/test-file-3
# md5sum /tmp/test-file-1
88c16a56754e0f17a93d269ae74dde9b  /tmp/test-file-1
# md5sum /tmp/test-file-2
db06069ef1c9f40986ffa06db4fe8fd7  /tmp/test-file-2
# md5sum /tmp/test-file-3
95227e10e2c33771e1c1379b17330c86  /tmp/test-file-3

Archive Zone testing

 We have our client’s ready, let’s check out the archive zone.

Create a new bucket and verify the bucket has been created in all RGW zones:

# s3apizone1 create-bucket --bucket my-bucket
# s3apizone1 list-buckets
BUCKETS 2023-03-15T12:03:54.315000+00:00        my-bucket
OWNER   S3 user to test the archive zone        archuser
# s3apiarchive list-buckets
BUCKETS 2023-03-15T12:03:54.315000+00:00        my-bucket
OWNER   S3 user to test the archive zone        archuser

Verify that the object versioning is not yet configured as this is implemented lazily

# s3apizone1 get-bucket-versioning --bucket my-bucket

# s3apiarchive get-bucket-versioning --bucket my-bucket

Upload a new object to our bucket my-bucket and verify the object has been created in all RGW zones.

# rclone copy /tmp/test-file-1 zone1:my-bucket

Verify how S3 versioning has been enabled in the archive zone but not in zone1:

# s3apiarchive get-bucket-versioning --bucket my-bucket
{
    "Status": "Enabled",
    "MFADelete": "Disabled"
}
# s3apizone1 get-bucket-versioning --bucket my-bucket

Verify that the object version ID is null in the master and secondary zones but not in the archive zone:

Modify the object in the master zone and verify a new version is created in the RGW archive zone.

Verify a new version has been created in the RGW archive zone:

# s3apiarchive list-object-versions --bucket my-bucket
{
    "Versions": [
        {
            "ETag": "\"db06069ef1c9f40986ffa06db4fe8fd7\"",
            "Size": 15,
            "StorageClass": "STANDARD",
            "Key": "test-file-1",
            "VersionId": "mXoINEnZsSCDNaWwCDELVysUbnMqNqx",
            "IsLatest": true,
            "LastModified": "2023-03-15T12:13:27.057000+00:00",
            "Owner": {
                "DisplayName": "S3 user to test the archive zone",
                "ID": "archuser"
            }
        },
        {
            "ETag": "\"88c16a56754e0f17a93d269ae74dde9b\"",
            "Size": 15,
            "StorageClass": "STANDARD",
            "Key": "test-file-1",
            "VersionId": "6DRlC7fKtpmkvHA9zknhFA87RjyilTV",
            "IsLatest": false,
            "LastModified": "2023-03-15T12:07:12.914000+00:00",
            "Owner": {
                "DisplayName": "S3 user to test the archive zone",
                "ID": "archuser"
            }
        }
    ]
}

You can check the ETag it will match the MD5sum for the file, this is only the case if multipart upload or object encryption is not configured.

# md5sum /tmp/test-file-2
db06069ef1c9f40986ffa06db4fe8fd7  /tmp/test-file-2
# md5sum /tmp/test-file-1
88c16a56754e0f17a93d269ae74dde9b  /tmp/test-file-1

Recovering S3 objects from the RGW archive zone

Let’s update one more version of the object

# rclone copyto /tmp/test-file-3 zone1:my-bucket/test-file-1

On the primary zone we only have one version, the current version of the object

# rclone --s3-versions lsl zone1:my-bucket

       15 2023-03-15 07:59:10.779573336 test-file-1

But in the Archive zone we have all 3 versions available:

# rclone --s3-versions lsl archive:my-bucket
       15 2023-03-15 07:59:10.779573336 test-file-1
       15 2023-03-15 07:59:03.782438991 test-file-1-v2023-03-15-121327-057
       15 2023-03-15 07:58:58.135330567 test-file-1-v2023-03-15-120712-914

So let’s delete test-file1 from my-bucket in zone1, and recover the object from the archive zone:

# rclone delete zone1:my-bucket/test-file-1
# rclone --s3-versions lsl zone1:my-bucket
# rclone --s3-versions lsl archive:my-bucket
       15 2023-03-15 07:59:10.779573336 test-file-1
       15 2023-03-15 07:59:03.782438991 test-file-1-v2023-03-15-121327-057
       15 2023-03-15 07:58:58.135330567 test-file-1-v2023-03-15-120712-914

The object has been delete from zone1, but still available in the archive zone with all it’s versions, if we recover the latest version test-file-1 it should match with the md5 for out test-file-3.

# rclone copyto archive:my-bucket/test-file-1 zone1:my-bucket/test-file-1
# rclone copyto zone1:my-bucket/test-file-1 /tmp/recovered-file1
# md5sum /tmp/recovered-file1
95227e10e2c33771e1c1379b17330c86  /tmp/recovered-file1
# md5sum /tmp/test-file-3
95227e10e2c33771e1c1379b17330c86  /tmp/test-file-3

But let’s say that we want to recover the object with the version that has a specific date like for example “2023-03-15-121327-057”.

# rclone --s3-versions copyto archive:my-bucket/test-file-1-v2023-03-15-121327-057 zone1:my-bucket/test-file-1
# rclone copyto zone1:my-bucket/test-file-1 /tmp/recovered-file1
# md5sum /tmp/recovered-file1
db06069ef1c9f40986ffa06db4fe8fd7  /tmp/recovered-file1
# md5sum /tmp/test-file-2
db06069ef1c9f40986ffa06db4fe8fd7  /tmp/test-file-2

This takes us to the end of the hands-on example of working with the archive zone and, with the help of clone, seamlessly recovering data from the archive zone.

Summary & up next

#Featured-area-2
#Featured-area-2-home
#Highlights
#Highlights-home