Originally posted by: Ulf Troppens
By Markus Rohwedder, Przemyslaw Podfigurny, Stefan Roth and Ulf Troppens
IBM Spectrum Scale provides means to include a broad range of storage devices into the same file system and to automate their efficient use based on policies. Spectrum Scale allows to group storage devices based on their performance, cost, locality and reliability characteristics (for example SSD drives, spinning disk drives or tape storage) in so called storage pools. A Spectrum Scale file system comprises one or more storage pools. Spectrum Scale also includes a policy engine which quickly identifies files based on their attributes and manages them automatically via rules.
Spectrum Scale rules enable automation of file placement, migration, listing, compression, encryption and deletion. A set of rules is called policy. The active policy of a file system must contain a default placement rule. Additionally it can contain further rules to handle the placement of selected files based on criteria (e.g. file type) and migration, compression and deletion rules which start certain actions based on capacity usage thresholds.
Properly configured rules optimize the use of premium and less expensive storage resources. In this blog posting we describe basic policies for the placement and movement of files:
- File placement rules enable the automatic placement of new files in a specific storage pool.
- File management rules enable the automatic file management during their life cycle, for instance by moving them from one storage pools to another, copying them to archival storage, changing their replication status, compressing or deleting them.
Spectrum Scale GUI supports the management of file placement and file management rules for the most common use cases. More sophisticated use cases can be configured with the Command Line Interface (CLI).
To open the GUI panel for policies and rules go to Files → Information Lifecycle.
The example shows a file system 'fs1' which is pre-configured with two storage pools (‘system’, ‘Archive’). Each file system includes the system storage pool and optional additional storage pools. Metadata is stored in the system storage pool only. In addition, the system storage pool can optionally contain data. For this example we assume that the system storage pool includes SSD drives to provide best performance for metadata and data. The ‘Archive’ storage pool uses near-line SAS disk drives. The active policy includes two placement rules (‘mp3Placement’, ‘default’) and one migration rule (‘thresholdMigration’).
The ‘mp3placement’ rule applies to all files which end with ‘.mp3’. The idea is that mp3 files never need to be stored on expensive SSD drives. They directly get stored on the near-line SAS disk drives in the Archive pool, when they are created.
Spectrum Scale stores rules in a SQL-like syntax. The GUI generates the code for the rules and activates it. Here is the syntax for the ‘mp3placement’ rule:
In our example the 'second' placement rule is named 'default'. It stores all new files in the ‘system’ storage pool. The rule ordering is essential, because Spectrum Scale searches for matching rules top down. The first matching rule will be applied and the default rule will be used only, if no other rule matches before. Therefore, the default placement rule needs to be placed after all other placement rules.
Here is the syntax for the ‘default’ rule:
In contrast to placement rules which are only evaluated during file creation, a migration rule relates to existing files. In our case the migration rule is evaluated only when a certain capacity usage threshold is reached. The ‘thresholdMigration’ rule starts to migrate files from the ‘system’ storage pool to the ‘Archive’ storage pool, once the ‘system’ storage pool is filled to 80%. As migration priority we have decided to migrate cold files defined by their last access time first. Please note that we never migrate files owned by user ‘postgres’, because we want to keep database files always on SSD drives. The migration already stops when 60% are reached because we want to keep as much data on SSD drives as possible.
Here is the syntax for the ‘thresholdMigration’ rule:
Spectrum Scale policies are a powerful tool for automated file management. This blog posting gives just a basic introduction. Check out Spectrum Scale knowledge center for more details or download the evaluation virtual machine and try it on your own.
References
IBM Spectrum Scale 4.2 Knowledge Center
http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_policyrules.htm
Spectrum Scale Evaluation Virtual Appliance
https://www.ibm.com/developerworks/servicemanagement/tc/gpfs/evaluate.html