Primary Storage

 View Only

Using IBM Spectrum Scale Policies for automated Information Lifecycle Management

By Archive User posted Wed December 16, 2015 05:14 AM

  

Originally posted by: Ulf Troppens


By Markus Rohwedder, Przemyslaw Podfigurny, Stefan Roth and Ulf Troppens
 

IBM Spectrum Scale provides means to include a broad range of storage devices into the same file system and to automate their efficient use based on policies. Spectrum Scale allows to group storage devices based on their performance, cost, locality and reliability characteristics (for example SSD drives, spinning disk drives or tape storage) in so called storage pools. A Spectrum Scale file system comprises one or more storage pools. Spectrum Scale also includes a policy engine which quickly identifies files based on their attributes and manages them automatically via rules.

Spectrum Scale rules enable automation of file placement, migration, listing, compression, encryption and deletion. A set of rules is called policy. The active policy of a file system must contain a default placement rule. Additionally it can contain further rules to handle the placement of selected files based on criteria (e.g. file type) and migration, compression and deletion rules which start certain actions based on capacity usage thresholds.

Properly configured rules optimize the use of premium and less expensive storage resources. In this blog posting we describe basic policies for the placement and movement of files:

  • File placement rules enable the automatic placement of new files in a specific storage pool.
  • File management rules enable the automatic file management during their life cycle, for instance by moving them from one storage pools to another, copying them to archival storage, changing their replication status, compressing or deleting them.

Spectrum Scale GUI supports the management of file placement and file management rules for the most common use cases. More sophisticated use cases can be configured with the Command Line Interface (CLI).
 

To open the GUI panel for policies and rules go to Files → Information Lifecycle.

image

 

The example shows a file system 'fs1' which is pre-configured with two storage pools (‘system’, ‘Archive’). Each file system includes the system storage pool and optional additional storage pools. Metadata is stored in the system storage pool only. In addition, the system storage pool can optionally contain data. For this example we assume that the system storage pool includes SSD drives to provide best performance for metadata and data. The ‘Archive’ storage pool uses near-line SAS disk drives. The active policy includes two placement rules (‘mp3Placement’, ‘default’) and one migration rule (‘thresholdMigration’).

image

 

The ‘mp3placement’ rule applies to all files which end with ‘.mp3’. The idea is that mp3 files never need to be stored on expensive SSD drives. They directly get stored on the near-line SAS disk drives in the Archive pool, when they are created.

image

Spectrum Scale stores rules in a SQL-like syntax. The GUI generates the code for the rules and activates it. Here is the syntax for the ‘mp3placement’ rule:

image

 

In our example the 'second' placement rule is named 'default'. It stores all new files in the ‘system’ storage pool. The rule ordering is essential, because Spectrum Scale searches for matching rules top down. The first matching rule will be applied and the default rule will be used only, if no other rule matches before. Therefore, the default placement rule needs to be placed after all other placement rules.

image

Here is the syntax for the ‘default’ rule:

image

 

In contrast to placement rules which are only evaluated during file creation, a migration rule relates to existing files. In our case the migration rule is evaluated only when a certain capacity usage threshold is reached. The ‘thresholdMigration’ rule starts to migrate files from the ‘system’ storage pool to the ‘Archive’ storage pool, once the ‘system’ storage pool is filled to 80%. As migration priority we have decided to migrate cold files defined by their last access time first. Please note that we never migrate files owned by user ‘postgres’, because we want to keep database files always on SSD drives. The migration already stops when 60% are reached because we want to keep as much data on SSD drives as possible.

image

Here is the syntax for the ‘thresholdMigration’ rule:

image

 

Spectrum Scale policies are a powerful tool for automated file management. This blog posting gives just a basic introduction. Check out Spectrum Scale knowledge center for more details or download the evaluation virtual machine and try it on your own.


References

IBM Spectrum Scale 4.2 Knowledge Center
http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_policyrules.htm

Spectrum Scale Evaluation Virtual Appliance
https://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html

3 comments
8 views

Permalink

Comments

Tue July 12, 2016 10:31 AM

Originally posted by: Nils Haustein


The link to the ILM Policy whitepaper has changed, here is the new one: http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102642 Sorry about the inconvenience.

Mon June 20, 2016 08:31 AM

Originally posted by: Andreas_Landhaeusser


http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102518 link doesn't exist anymore :-( any info how to get additional info about querying metadata and the known tags to query regards Andreas

Thu December 17, 2015 04:12 AM

Originally posted by: Achim-Christ


For further information on Spectrum Scale ILM policies, along with additional meaningful samples, you can also refer to the following whitepaper: IBM Spectrum Scale™ (GPFS) Archiving Policies http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102518