Introduction

A tiered storage system provides lower total cost of ownership for large volumes of data by storing data on the most appropriate storage tier (flash, disk and tape). Independent studies have demonstrated that total cost of ownership of tape solution provides an expected TCO that is more than 80% lower than that of the all-disk solution [1].

While tape storage is suitable for storing large volumes of data over long periods of time at lower cost, access time to data on tape is significantly higher than to data on disk. This can cause negative user experience which is amplified by standard file systems like NFS and SMB that do not provide functions to efficiently manage data on tape. Many users of tiered storage file systems have experienced the implications when using tapes and are looking for alternatives.

This article explains the architecture and operations in tiered storage file systems with tape (see section Tiered storage file systems with tapes) and highlights the challenges with tapes (see section Challenges). Based on the challenges best practices are derived and explained (see section Best practices). Finally, solutions addressing the challenges and implementing best practices are described (see section Solutions). 

Tiered storage file systems with tapes

A tiered storage file system as shown in the picture below, provides disk and tape storage within a global file system namespace where a space management component transparently moves files from disk to tape and vice versa. The file system name space is accessible by users and applications through standard file system protocols such as NFS, SMB and POSIX.

Tiered storage file systems do not always include tape storage. In this article however we address the challenges and solutions when using tape storage. Tiered storage file systems with tape storage are primarily used for archiving large volumes of data for long periods of time.

Tiered storage file systems can also be accessed through object storage protocols such as AWS S3 provided by object storage services using the tiered storage file system as back-end storage.

Tape operations

The movement of files to tape is called migration. During migration the file is copied to tape and subsequently stubbed on disk. The file stub on disk is represented by the file’s inode including metadata about the file such as time stamps, extended attributes, size, path and file name and a reference to the tape-ID where the file is stored. The file data however is freed up on disk and resides on tape. The space management component uses policies to automatically migrate files between disk and tape. These policies can select files subject for migration based on file metadata and other attributes.

A special form of migration is pre-migration. With pre-migration the file is dual resident: on disk and on tape. Hence pre-migration only copies the file to tape but does not stub it. The advantage of pre-migration is that a subsequent migration is fast because the file does not have to be copied to tape again, it is just stubbed on disk. Pre-migration however does not free up space on disk.

A migrated file is visible within the file system name space and can be transparently accessed by the users and applications. When a migrated file is accessed then the space management component intercepts this access request and copies the file back from tape to disk. Once the file is back on disk the access request is granted. This process is named recall. There are two kinds of recalls: transparent recall and tape optimized recall.

A transparent recall is triggered when a migrated file is transparently accessed in the file system. The space management component loads the tape-ID referenced in the file stub, locates the file on tape and reads the file back to disk. When multiple migrated files are accessed at the same time, then multiple independent transparent recalls are triggered, each requiring a tape to be loaded, located and a file to be read. If multiple files are on the same tape, then these files are not recalled in the order they are stored on tape. This causes tape start – stop operations which are usually time consuming. In addition, transparent recalls can cause chaotic tape load operations because files are not sorted by their tape-ID. This is even more time consuming.

Tape optimized recalls are much faster and tape resource gentle. The tape optimized recall is triggered by a command that includes a list of file names to be recalled. This command is passed to the space management component that sorts these file names by the tape-ID (located in the file stub) and the position on tape. Afterwards it mounts the required tapes and copies the files back to disk in parallel. Hence, all files that are on one tape are copied back in the order they are stored on tape. In addition, it prevents chaotic tape loads because each tape that is required is loaded and processed only once. The downside is that tape optimized recalls in a tiered storage file system are not transparent to the user and require an administrative command to be issued.

Challenges

Tiered storage file systems with tape storage are a blessing and a curse. The blessing is that the user can see all files regardless if these are stored on disk or tape. Cursing starts when the when the user opens a file that is stored on tape because the recall takes one or more minutes. Unfortunately the user is not aware that the file is on tape because standard file systems do not indicate whether the file is on disk or on tape. The user must wait and does not know why.

It gets even worse if the user simultaneously opens several migrated files. Each access to a migrated file causes the recall of a single file. Accessing hundreds of migrated files causes hundreds of recalls, each recall process copies a single file from tape to disk. This causes even longer waiting times because transparent recalls are not tape optimized. Standard file systems cannot leverage tape optimized recalls where files are sorted by their tape-ID and location on tape before the recall is initiated in this order.

These challenges are amplified by standard file systems that make the user think all files are instantly available but access to files on tape takes time. Making standard file systems tape aware is not feasible because it requires changes in hundreds of heterogeneous applications, file system implementations and operating systems that use standard file systems according to the current specification.

Some operating system tools also cause transparent recalls without the user noticing it. For example, the Finder on MacOS causes transparent recalls when it accesses a tiered storage file system via SMB because this it opens all files in a certain directory to read the thumbnail information. Opening a migrated file causes transparent recalls and the Finder can cause recall storms.

To address these challenges, architectures and operational process supported by tools can be established to facilitate archiving and retrieval in a tiered storage file system. The next section Best practices provides guidance for architecting tiered storage file systems with tape and presents process for archival and retrieval as well as a Tape Archive REST API tool to support these processes. We also reference object storage solutions that have built in tape awareness.

Best practices

Using tapes within tiered storage file systems requires some fundamental architectural decisions combined with operational process to overcome the challenges with tape. Find below some key recommendations and high-level guidance for implementing operational process.

General guidance

This section highlights key recommendations for implementing tiered storage solutions with tape.

• Storing small volumes of data on tape will not save cost compared to disk storage because the initial investment requires a tape library, tape drives and tape cartridges. The cost for this is comparable to disk storage.
• Storing data on tape for short period of time requires frequent reclamation of tapes which is a resource intensive process because it requires two tape drives.

• Accessing data on tape takes long time for the recall
• Recalls scale with the number of tape drives. As more recalls as more tape drives are required which increases cost

• The disk buffer in the archive file system is the landing zone for all incoming archived data. It assures that the user does not directly access the tape resources during archiving.
• Provide enough capacity to the disk buffer assuring that migration to tape takes place after the ingest process.

• Using tape within a production file system may not be feasible because of frequent access to data and the high access latency associated with accessing data on tape.
• Separating the archive file system with tape from the production file system without tape allows different file system configuration parameters and operational processes on a file system level

• If the user can browse the archive file system, then he can see all his archived files which makes him happy. 
• Preventing transparent recalls mitigates the challenges with tape and requires additional operational processes for retrieval. Further considerations about preventing transparent recalls can be found in section Prevent transparent recalls.

• Archival process facilitates the ingest of files into the archive file system and the migration to tape.
• Retrieval process facilitates the retrieval of data from the archive file system and the optimized recall from tape.
• Archival and retrieval process must be simple to allow users with different IT skills to follow it.
• More recommendations regarding these process see sections Archival process and Retrieval process.

Prevent transparent recalls

The objective is to prevent the user from triggering a transparent recall when accessing a file. This avoids long waiting times and recall storms. However, it is disruptive because the user cannot just open a migrated file (see section Retrieval process. There are different ways to prevent transparent recalls.

One way is based on the capabilities of the storage Application Programming Interface (API) in combination with the storage system. For example, AWS S3 Glacier API does not allow to GET objects in storage class GLACIER. The user must issue a restore request for the object using the S3 Glacier API before the object can be accessed. For more information see section AWS S3 Glacier  Unfortunately, file system interfaces like Network File System (NFS) or Server Message Block (SMB) do not provide this functionality.

Another way to prevent transparent recalls is based on the capabilities of the space management component. If the space management component allows for preventing transparent recalls, then transparent recalls cannot be executed, and tape optimized recalls can be done.

Yet another ways to prevent transparent recalls of migrated files is to remove permission for read and write for the user and group after the file has been migrated to tape. The implementation requires to periodically identify files that are migrated and remove read and write permissions for the user. Likewise, for migrated files that have been recalled the read and write permissions must be restored to facilitate access. Automated processes can be used to identify migrated and recalled files and adjust the file permissions. However, this adjustment of permissions is asynchronous and does not entirely prevent transparent recalls.

The prevention of transparent recalls is the foundation for the archival and retrieval processes discussed below. 

Archival process

The archival process defines how files are moved to the archive file system and when they are migrated to tape. The user can access the archive file system that should be separated from the production file system and should provide a disk buffer as landing zone for the incoming data. This assures that the user does not send the archive data directly to tape.

A simple implementation of an archival process is that all users mount the archive file system and whenever a project is finished the user moves the associated files and directories to the archive file system. This way the user is actively involved in the archival process and knows that his data is in the archive file system.

The subsequent migration of files to tape should not require user intervention, just to make sure the archival process is simple. The migration can be automated and policy-drive. Policies define the selection criteria for files to be migrated. These selection criteria should match the expected access pattern of files in the archive file system. For example, if the expected access pattern is that files are more often accessed within the first 30 days after archival then files should be selected for migration if the file access time is greater than 30 days. This assures that files that have been migrated to tape are rarely accessed.

In some instances, it might be useful if the user triggers the migration to tape. For example, if the file access pattern is specific and cannot be implemented in automated policies. In this case the user can leverage extra tools like the Tape archive REST API to order the migration to tape (see section Tape archive REST API).

Once files are archived in the archive file system the user can see all his files and follow the retrieval process if he needs to access files. 

Retrieval process

The retrieval process defines how files are retrieved by the user from the archive file system. The user has access to the archive file system and can see all the file he has archived. One important prerequisite is that the user cannot transparently access a file that is migrated, see section Prevent transparent recalls for more details.

Preventing transparent recalls is disruptive and requires additional operational processes comprised of the following steps:

·        The user checks if a file is migrated, either by evaluating file or object attributes or by attempting to open the file. If transparent recalls are prevented, then the user cannot open the file.

·        The user signals the names of the migrated files to be recalled.

·        The files to be recalled from many users are collected and recalled in a tape optimized manner. The time interval for the tape optimized recalls can be defined using service level agreements.

·        The user checks if the file was recalled and if this is the case, then the user opens the file normally.

Preventing transparent recalls may require establishing service level agreement (SLA) defining how long the user must wait before he can access a migrated file. This defines the time interval of tape optimized recall operations. The definition of such SLA is inherent part of the retrieval process. One example for such an SLA is that the user must wait 2 hours before a migrated file has been recalled using tape optimized recalls.

The next chapter presents solutions implementing the best practices with focus on preventing transparent recalls and executing tape optimized recalls instead.

Solutions

This chapter demonstrates solutions allowing to implement the best practices for tiered storage systems with tape.

AWS S3 Glacier

With AWS S3 Glacier recall storms can be prevented. AWS S3 Glacier is an addition to the AWS S3 API providing operations that facilitate asynchronous retrieval or restoration of objects from high latency media like tapes [5]. AWS S3 Glacier works in combination with the underlying storage system capabilities to automatically perform tape optimized recalls.

The picture below gives an example of an S3 object storage service in combination with tiered storage file system:

As shown in the picture above, the user uses the S3 API to interact with the system. The system provides a S3 Glacier object storage service. Objects are stored in the underlying tiered storage file system that is space managed by the space management component. The space management component automatically migrates objects and recalls objects in a tape optimized manner. 

Object stored in a Glacier storage class cannot be retrieved using standard GET operations. Instead, the S3 user must first use the Restore-Object operation provided by the Glacier API to request the recall from tape. Upon retrieval of the restore request, the storage system queues the file associated with the object for recall and eventually recalls multiple objects by leveraging tape optimized recalls. Once the object was recalled, the object metadata reflects that the object can be GET. Now the user can issue a normal GET operation to access the object.

Find below a short example demonstrating S3 Glacier in action along with NooBaa running on IBM Storage Scale [6]:

The S3 user copies object into a bucket and associates the storage class GLACIER with the object:

The S3 user uses the head-object operation to show the object metadata. The object metadata indicates that the object is in the storage class: GLACIER:

The S3 user attempts to GET the object. This operation fails because the objects in storage class GLACIER:

Instead, the S3 user must trigger a restoration of the object using the restore-object operation:

The parameter –restore-request Days=1 means that after the file was restored (recalled) it remains 1 day on disk, before it is re-migrated to tape.

The subsequent head-object operation shows that the object is scheduled for restore:

The tag ongoing-request=true means that the object must be recalled from tape. NooBaa stores the tag ongoing-request=true as attribute of the file in the IBM Storage Scale file system.

At this point the underlying storage system performs the tape optimized recall of all objects that have a restore request (ongoing-request=true). With IBM Storage Scale, the policy engine can be used to identify all objects that have the attributed ongoing-request=true and recall these objects using the tape optimized recall function. The tape optimized recall for bulks of objects automated and executed at the time intervals that are the foundations of the service level agreements.

After the object was recalled from tape, the S3 user issues a head-object call to see if the object can be accessed:

The tag ongoing-request is set to false, and the tag expiry-date is in the future, indicating that the object can be retrieved.

The S3 user can now copy the object from the S3 Glacier storage:

Finally, the storage system automatically re-migrates the object to tape. For this the IBM Storage Scale policy engine can be used to identify objects, where the expiry-date is expired, and migrate bulks of identified objects to tape.

AWS S3 Glacier combines the S3 API with capabilities of the storage system and provides tape aware object storage. It implements many best practices (see section Best practices):

• Avoids recall storms because the S3 user does trigger transparent recalls.
• The user can determine if an object is on tape or not.
• The user can determine if the object can be accessed using the S3 GET function by using the head-object function.
• The user can control which objects must be recalled using the restore-object function.
• Restore requests from multiple users are consolidated and the objects are recalled in a tape optimized manner. 
• Service level agreements defining the time between the restore request and the accessibility of the objects can be implemented by using schedules for the tape optimized recalls.

Integration with metadata management system

Another way to address challenges of tiered storage file systems with tape can be accomplished with metadata management systems. A metadata management system automatically and continuously catalogues the system metadata of files stored in a tiered storage file system. From this metadata repository the user can easily determine if a file is stored on tape. The user can add additional metadata to the metadata management system that can be used to recall files in an optimized manner [7].

The picture below gives an overview of the integration of metadata management system with a tiered storage file system:

Files are stored in the tiered storage file system through file and object storage protocols. The space management component manages migration and recalls of files on tape. 

The metadata management system catalogues the metadata of the files in the tiered storage file system. The metadata of files in a file system includes file and path names, time stamps, file size, storage tier, migration state, permissions, and other attributes. The metadata management system allows the user to view the metadata of files. In addition the user can add metadata to the files within the metadata management system. The interaction between the user and the metadata management system is done via a GUI or REST API.

Using the metadata management system, the user gets a global view of the files and the associated metadata, including the migration state. With this the user knows if a file is on tape or not.

To avoid recall storms, the space management component is configured to prevent transparent recalls.

Access to files on tape works like this:

• When the user accesses a file that is on tape, then the space management component prevents the transparent recall. The user gets an error from the file system.
• The user queries the file state via the metadata management system and determines that the file is on tape.
• The user adds an additional tag (recall-tag) to the file metadata indicating that the file must be recalled. 
• An automated process running within the tiered storage file system periodically queries the metadata management system for files with the recall-tag set and recalls these files using the tape optimized recall.
• After the recall completed, the automated process adjusts the migration state of the file and  recall-tag in the data management system.
• Within the defined period the user checks the file state via the metadata management system and determines that the file is on disk.
• The user can now access the file. 
• Later, files are re-migrated again.

Metadata management system in combination with tiered storage files system provide tape aware storage. It implements many best practices (see section Best practices):

• Avoids recall storms because transparent recalls are disabled.
• The user can determine if a file is on tape by querying the file metadata in the metadata management systems.
• The user can control which files must be recalled using by adding the recall-tag to the metadata of the file in the metadata management system.
• Files with the recall-tag from multiple users are consolidated and the files are recalled in a tape optimized manner. 
• Service level agreements defining the time the user must wait until the files tagged for recall are recalled can be implemented by using schedules for the tape optimized recalls.

Tape archive REST API

The Tape archive REST API is on open-source project that facilitates tape optimized operations in an archive file system [2] by giving control to the user for managing his files. It can be used to support the archival and retrieval process.  The Tape archive REST API introduces an additional control path beside the existing data path as shown in the picture below:

As shown in the picture above, the user reads and writes files in the tiered storage file system through the data path, leveraging standardized file and object storage protocols. In addition to the data path, there is a control path provided by the Tape archive REST API service. This service provides endpoints for the user in the front end and performs tape optimized operations in the backend, by interacting with the space management component. 

The efficient use of the Tape archive REST API requires that transparent recalls are prevented.

The current implementation of the REST API is based on IBM Storage Scale providing the archive file system and IBM Spectrum Archive Enterprise Edition version 1.3.3.4 as space management component. The Tape archive REST API is not part of an IBM products and can be used under the open-source license without support. Conceptual this REST API can be adapted for many types of archive file systems and space management components. Contact the author of this article if you need professional help to fit the API to your requirements and environment.

In the next section typical use cases for the Tape archive REST API are described.

The API in action

A typical use case is that the user can see all his files through an NFS export, a SMB share or directly in the Spectrum Scale file system. Access to files that are migrated is not possible, because transparent recalls are prevented. Preventing transparent recalls requires additional measures (see section Prevent transparent recalls).

If the access to a file fails than the user can determine the file state using the following API-call:

The fully qualified path and file name is given within the URL. As shown in the Response of this call, the file is in state “migrated”.

Because transparent recalls should be prevented the user can now use the following API-call to request recall of the files:

The fully qualified path and file name(s) are given in the body of this request, whereby each path and file name must be placed in a separate line. The response indicates that the recall has succeeded. To leverage the tape optimized recall multiple file name should be given with the recall request.

The user can now determine the file state again using the appropriate API-call:

As shown in the response the file is in state “premigrated”.

The user can also order the migration of a pre-migrated or resident file using the following API call:

The URL identifier “pool1=pool1” is required and defines the target tape pool for this migration request. Up to three pools can be defined as migration target. The fully qualified path and file name(s) are given in the body of this request, whereby each path and file name must be placed in a separate line. The response of this request shows that the migrate has succeeded. For tape optimized operations multiple file names should be given with this migrate request.

The Tape archive REST API provides additional function to inquire the status of certain IBM Spectrum Archive EE components. It can be deployed on IBM Spectrum Archive EE servers or on remote servers that communicate with the IBM Spectrum Archive EE servers via ssh. It can even be deployed as docker container. For more information refer to the git repository [2].

OpenStack Swift HLM

Many of the challenges associated with tape storage in a tiered storage file system are addressed with OpenStack Swift High Latency Middleware (Swift HLM). The OpenStack Swift HLM middleware is an OpenStack Swift associated project [3] and is useful for tiered storage file system with tape. OpenStack Swift HLM is designed to manage operations on a high latency media like tapes [4]. OpenStack Swift HLM allows explicit control of Swift objects and container locations by providing tape specific functions for migration, recall and status. Architecturally, OpenStack Swift with HLM incorporates the data path and the control path in one protocol as shown in the picture below:

As shown in the picture above, the user leverages the OpenStack Swift and HLM API to communicate with the OpenStack Swift and HLM object storage service. OpenStack Swift provides the data path to manage objects. OpenStack Swift HLM provides the control path to manage tape optimized operations. OpenStack Swift HLM extends the OpenStack Swift API with additional endpoint modifiers, such as:

• With the “status” call, the user can easily determine if objects are stored on tape.
• With the “migrate” call, the user can initiate the migration of an object or container.
• With the “recall” call, the user can initiate recalls for migrated object or containers by leveraging tape optimized recalls.

To avoid recall storms, the space management components must be configured to prevent transparent recalls.

Similar, to AWS S3 Glacier, OpenStack Swift HLM provides control over the migration and recall operations, while OpenStack Swift provides the object storage services like AWS S3. Unfortunately, the acceptance of OpenStack Swift in the market is declining in favor of AWS S3.

References

[1] Disk and Tape TCO study by ESG:
https://www.lto.org/wp-content/uploads/2018/08/ESG-Economic-Validation-Summary.pdf

[2] Github repository for Tape archive REST API
https://github.com/nhaustein/tape-archive-api

[3] OpenStack Swift associated projects providing alternative APIs:
http://docs.openstack.org/developer/swift/associated_projects.html#alternative-api

[4] OpenStack Swift HLM
https://developer.ibm.com/open/projects/swift-high-latency-media/

[5] Amazon S3 Glacier
https://aws.amazon.com/de/glacier/

[6] Using AWS S3 glacier with NooBaa
https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2024/03/22/s3-tiering-to-tape-with-noobaa-part-5-glacier

[7] Using metadata management systems to address tiered storage challenges
https://community.ibm.com/community/user/storage/blogs/nils-haustein1/2022/02/10/spectrumdiscover-taperecall-integration