File and Object Storage

In part 3 of this article series, we explained basic techniques to tier (migrate and recall) S3 objects from disk to tape using NooBaa, IBM Storage Scale and IBM Storage Archive EE. In part 4, we demonstrated techniques leveraging object metadata and tags to better control migrations and recalls. Object metadata and tagging requires discipline from the S3 user to apply these metadata fields and tags consequently. Unless transparent recalls are disabled using Storage Archive options, there is still the danger of getting many transparent recalls that can lead to recall storms.

To avoid recall storms, NooBaa supports the AWS S3 Glacier API.  Glacier provides additional operations on top of the S3 API facilitating retrieval or restoration of objects from high latency media like tapes. Object stored in a Glacier storage class cannot be retrieved using standard GET operations. Instead, the S3 user must first use the Restore-Object operation provided by the Glacier API to request the recall from tape. After the object was recalled from tape to disk, the user can use the standard GET operations to retrieve the object.

In this part of the blog article series, we explain the fundamentals of AWS S3 Glacier and demonstrate how it works with NooBaa in combination with Storage Scale. We do not migrate and recall objects during this demonstration because it requires manual interventions. We will elaborate on automation of migration and recalls along with Glacier operations in the next part of this series.

NooBaa Glacier fundamentals

In this section we demonstrate the fundamental behavior of Glacier with NooBaa on IBM Storage Scale. We show manual steps to prepare Glacier objects for restoration to foster the understanding of the Glacier implementation with NooBaa NSFS. In the next article, we will demonstrate automation for all many manual steps shown below.

Enable Glacier in NooBaa

To enable the basic Glacier function in NooBaa, two parameters must be added to the NooBaa configuration file (for more details about the NooBaa configuration file refer to Section Customize and start the NooBaa service of Part 2 in this series). The two configuration parameters are NSFS_GLACIER_ENABLED=true and NSFS_GLACIER_LOGS_ENABLED=false as shown in the example below:

Note, setting NSFS_GLACIER_LOGS_ENABLED to false will not log the objects to be migrated or recalled in a log file. We use this option to demonstrate manual steps. When automating migration and recalls of Glacier objects, this option is set to true.

If the parameter NSFS_GLACIER_LOGS_ENABLED is set to true (default) or when running a downstream version of NooBaa, then an additional directory must be created. Check if these the following directory exists:

If this directory does not exist, then create this directory and allow the user running the NooBaa service read and write access:

To make the new configuration parameter effective the NooBaa service must be restarted:

Objects are associated with Glacier through the storage class GLACIER. The storage class is provided during PUT operation by the S3 user. Objects associated with the storage class GLACIER cannot be GET without having them restored.

Store and migrate Glacier objects

As S3 user, let’s create a new bucket glacier1 where we store the objects associated with the storage class GLACIER:

Now, we PUT an object into the new bucket and associate the storage class GLACIER:

To determine the storage class of an object the s3 API head-object operation can be used:

The objects in storage class GLACIER can be migrated by using the Storage Scale policy engine. The file associated with the object has a user attribute user.storage_class set to GLACIER as shown below:

It is also possible to store non-Glacier objects in the bucket glacier1. This can be accomplished by omitting the parameter –storage-class GLACIER during the PUT operation. As a result, NooBaa sets the file attribute user.storage_class=STANDARD. Objects that are stored in the storage class STANDARD can be retrieved with normal GET operations.

The policy engine can be programmed to select files for migration that have the attribute user.storage_class set to GLACIER and that are not already migrated. Here is an example of this policy:

The storage admin executes this policy stored in file mig-glacier.policy by using the following command:

In the next section we elaborate on getting objects from Glacier.

Get Glacier objects

Normal GET operations for objects in storage class GLACIER do not work as shown below:

Instead, the S3 user must trigger a restoration of the object using the restore-object operation:

One day means that after the file was restored it remains 1 day on disk, before it is re-migrated to tape.

The subsequent head-object operation shows that the object is scheduled for restore:

The tag ongoing-request=true means that the object must be recalled from tape. Since we have not migrated the object, we must not perform a recall.

After the restore-object operation, the object is expected to be recalled from tape. We skip the recall for now because we did not migrate the object to tape. To enable the GET operation some attributes of the file in the Storage Scale file system must be adjusted. Before we adjust the attributes, let’s take a look at the current attributes of our Glacier object from a file system perspective. This requires administrative privileges in the Storage Scale file system:

As shown above NooBaa added two additional attributes: user.storage_class and user.noobaa.restore.request. The attribute user.storage_class was added when the object was PUT into the bucket. The attribute user.noobaa.restore.request was added when the restore-object operation was performed.

To signal to NooBaa Glacier that the file was recalled from tape, we must manually add a new attribute user.noobaa.restore.expiry that encodes the date and time when the file is being remigrated. To set this we take the 1 day period encoded in the attribute user.noobaa.restore.request into account. Setting attribute of a file must be executed by a Storage Scale administrative user with sufficient privileges:

Afterwards, we manually delete the attribute user.noobaa.restore.request as administrative user of the file system:

The file attributes look like this from a file system perspective:

As S3 user let’s look at the object using the head-object operation:

The tag ongoing-request is set to false, indicating that the object can be retrieved. The expiry time is in accordance to the file attribute user.noobaa.restore.expiry. The user must get the object prior to the expiration date, because afterwards the object might have been migrated again.

Let’s get the object:

The object can be remigrated to tape in accordance with the restore expiration time that specifies the date and time the object can reside on disk. After remigration, the file attribute user.noobaa.restore.expiry must be manually deleted. The example below shows how to delete this attribute from the file.

 This command must be executed by an administrative user of the Storage Scale file system that has permissions to manipulate the subject file metadata.

Subsequently the S3 user cannot retrieve the object using the normal GET operation:

The head-object operation shows that there is not restore request:

Summary

As shown above, NooBaa automatically adds extended attributes to the files stored in the Storage Scale file system in accordance with the S3 Glacier operation. These attributes can be used to control automatic migrating (user.storage_class=GLACIER) and recalls (user.noobaa.restore.request). To facilitate GET operations for Glacier objects, further attributes must be manually set (user.noobaa.restore.expiry) and deleted (user.noobaa.restore.request) after the recall. The table below summarizes the operations from an S3 user and storage admin perspective and shows the file attribute setting. The flow in this table is row-by-row with different operations highlighted in different colors:

Manually setting attributes after recall operations is cumbersome. Therefore, NooBaa provides a flexible plugins architecture to facilitate automatic migration and recalls. We will elaborate on this innovative architecture in our next part of this blog article series. Stay tuned!

References

[1] AWS Glacier API
https://docs.aws.amazon.com/amazonglacier/latest/dev/amazon-glacier-api.html

[2] AWS Glacier implementation with NooBaa
https://github.com/noobaa/noobaa-core/blob/master/docs/design/NSFSGlacierStorageClass.md