File and Object Storage

IBM Spectrum Scale policy example for file listing

By RAVIKUMAR NADAR posted Mon February 27, 2017 08:50 PM

  
IBM Spectrum Scale supports information lifecycle management(ILM) through a very powerful policy scan engine that enables the automation of file placement, migration, listing, compression,encryption and deletion . This is one of the most powerful and useful feature of IBM spectrum Scale as it can parallelly scan the the filesystem containing massive amount of files and directories in a very fast and efficient manner.
In fact policy engine have been instrumental in almost every new feature (AFM, TCT, Encrpytion, Compression, backup, archive etc ) supported by IBM Spectrum Scale
The policy engine supports a plethora of options on file selection based on it attributes and supports a familiar SQL like statement syntax for writing policy rule.
This blog explains one of the most common policy that might be useful in any deployment for the listing of files based on a file's creation, modification(POSIX mtime), access(POSIX atime) or change(POSIX ctime) time.

To list all the files created after a particular time, the following policy can be used

Policy file :

cat policy.txt
RULE 'filesRule' LIST 'files'
SHOW(varchar(mode) || ' ' || varchar(file_size))
WHERE (CREATION_TIME > TIMESTAMP('LAST_CREATE'))


Calling mmapplypolicy from the command line

$LC='2017-02-21 04:56 IST' #list all files created after this time
$LCU=$(date +%Y-%m-%d" "%H:%M -d "$LC" -u) #Timestamp has to be converted to UTC
$mmapplypolicy gpfs0 -P policy -I defer -f /tmp -M LAST_CREATE="$LCU"


The above policy will list all the files that were created after the timestamp "LAST_CREATE" that is passed using the -M parameter to mmapplypolicy. One important thing to consider here it that the timestamp "LAST_CREATE" should be in Coordinated Universal Time (UTC), since the file timestamp attributes are in local one has to convert the time used for the selection criteria to UTC as shown in the example above.

After the policy is run the list of files selected can be found in "list.files" in the directory specfied by the -f option

The the first 3 and last 2 columns of the output file is present by default for all listing , 4th and 5th column are the ones requested from the example policy above .

In this example following attributes are listed -

("Inode number" "gen number" "Snapshot ID" "mode" "File size" "Seperator" "Filename ")

Example listing 

more /tmp/list.files
57385 1566426600 0 -rw-r--r-- 5 -- /gpfs0/newfile
57387 1653928386 0 -rw-r--r-- 1048576 -- /gpfs0/file.31
57407 514130474 0 -rw-r--r-- 1048576 -- /gpfs0/file.39
57443 1997432655 0 -rw-r--r-- 1048576 -- /gpfs0/file.33
57445 1219835869 0 -rw-r--r-- 1048576 -- /gpfs0/file.38
57458 1777723776 0 -rw-r--r-- 1048576 -- /gpfs0/file.32


The "SHOW" clause above can be used to list any of the file attributes described in
https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adv_usngfileattrbts.htm
References

Spectrum Scale Information Lifecycle Management in Knowledge center
https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adv_storage_mgt.htm

White paper on Spectrum Scale Information Lifecycle Management with lots of examples
http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102642

Spectrum Scale Information Lifecycle Management using GUI
https://www.ibm.com/developerworks/community/blogs/storageneers/entry/Using_IBM_Spectrum_Scale_Policies_for_Automated_Information_Lifecycle_Management?lang=en

Few more file listing examples from Scott's blog
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Policy%20Examples
#timestamp
#Policy
#list
#IBMSpectrumScale
#Softwaredefinedstorage
#IBMSpectrumStorage
#ILM
#example
0 comments
6 views

Permalink