File and Object Storage

Spectrum Scale NAS at home part 8: Tuning and Quotas

By MAARTEN KREUGER posted 19 days ago

  

Spectrum Scale is a complex product for being just a filesystem, especially when you only use it on a single server. It's designed for many servers, multiple sites, and multiple filesystems. We're keeping it simple in this blog series on home NAS, so just one server, and one filesystem. Still, it's a good idea to have a little background information on how spectrum scale works and how you can tune it. Let's go!

The spectrum scale daemon (mmfsd) does ALL the work. we have some extra services around it, like mmsysmon(health),  gpfsgui, ssh, and SMB daemon, but core file system and cluster services are provided by the multi-threaded GPFS daemon over a single TCP/IP port, 1191. The cluster and filesystem configuration is stored in the /var/mmfs/gen/mmsdrfs file, with additional information stored in the Central Cluster Repository and other directories. (/var/mmfs/) You should change settings using the mmch* commands only:

  • mmchconfig : Change cluster settings, per node, or per nodeclass
  • mmchnode: Change node designations like manager, quorum, etc
  • mmchfs: Change filesystem settings like quota, default replication
  • mmchattr: Change individual file attributes like compression, replication

If you want to know what's happening in the cluster, that's where the mmdiag command comes in. A few helpful options are:

  • mmdiag --waiters : shows mmfsd threads that are waiting on something, possibly due to slow IO.
  • mmdiag --config : shows ALL the cluster settings for this node.
  • mmdiag --stats : shows cache utilisation

The GPFS filesystem cache is not like regular Linux cache, as the GPFS IO is processed in user space. You specify a fixed amount of memory for cache usage that we call the pagepool. This part of memory is used to store cached file data pages. Separate from this is the number of files cached (maxFilesToCache) and the file stats that may be cached (maxStatCache). For a full description, see:  https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=configuring-parameters-performance-tuning-optimization

 We'll set a few cluster wide parameters to do basic tuning:

# mmchconfig pagepool=1G
# mmchconfig maxFilesToCache=10000
# mmchconfig maxStatCache=20000

If you want to see which files are in file cache, use the mmcachectl command:

# mmcachectl show --show-filename

FSname  Fileset  Inode  SnapID   FileType     NumOpen  NumDirect  Size         Cached      Cached    FileName
------------------------------------------------------------------------------------------------------------------------------
nas1    0        4040   0        directory    0        0          3968         0           FD        /nas1/ces/ces/connections
nas1    1        66097  0        file         0        0          2614817792   939524096   F         /nas1/Documents/SpectrumArchive_1.3.0.5_TrialVM.ova

 
Each filesystem has its own settings, and most are functional. There are some important parameters that influence performance:

  • block size; cannot be changed afterwards, data is written to each disk in a pool with this maximum IO size
  • Replication; The max setting cannot be changed afterwards, but default values can. Set this to two, and you'll do twice the writes.
  • write-cache-threshold; If you have fast SSD for the system pool, smaller writes (max 64k) are stored in the Log, which also needs to be large enough. Look up HAWC in the manual for more information.

Not a performance related parameter, but important when managing a GPFS file system is the number of inodes (file/dir objects) that are defined and pre-defined. Different from most filesystems, GPFS a maximum setting for inodes per filesystem, and per fileset. You can change this value, and also pre-define inodes so new files can be created faster. You can see the current usage with the mmdf command:

# mmdf | tail -6
Inode Information
-----------------
Total number of used inodes in all Inode spaces:               4075
Total number of free inodes in all Inode spaces:             564245
Total number of allocated inodes in all Inode spaces:        568320
Total of Maximum number of inodes in all Inode spaces:       568320


 It's important to keep some inodes free, so if the number of free inodes runs low, you can change the values with mmchfs: 

# mmchfs nas1 --inode-limit=2M:1M
# mmlsfileset nas1 -L

Filesets in file system 'nas1':1
Name             Id      RootInode  ParentId Created                      InodeSpace      MaxInodes    AllocInodes Comment
root             0              3        -- Mon Apr 12 18:03:44 2021        0              2097152        1049600 root fileset
Documents        1           4042         0 Tue Apr 13 09:06:55 2021        0                    0              0
WinShare        2           4044         0 Tue Apr 13 09:20:40 2021        0                    0              0

 

For independent filesets, setting inodes is similar, you set the maximum number of inodes and a pre-allocated number of inodes and these are located in a separate inode space from the main filesystem. With dependent filesets inodes from the main inode space are always used. In this example I'll create a new independent fileset and then increase only the maximum inodes, as we still have plenty of pre-allocated inodes:

# mmcrfileset nas1 new-fileset --inode-space new
# mmlinkfileset nas1 new-fileset -J /nas1/new-fileset
Fileset new-fileset linked at /nas1/new-fileset
# mmlsfileset nas1 new-fileset -L
Filesets in file system 'nas1':
Name               Id      RootInode  ParentId Created                      InodeSpace      MaxInodes    AllocInodes Comment
new-fileset        5         786435         0 Mon Jul  5 15:01:01 2021     6              100352       100352

# mmchfileset nas1 new-fileset --inode-limit 200K
Set maxInodes for inode space 6 to 204800
Fileset new-fileset changed.
# mmlsfileset nas1 new-fileset -L
Filesets in file system 'nas1':
Name               Id      RootInode  ParentId Created                      InodeSpace      MaxInodes    AllocInodes Comment
new-fileset        5         786435         0 Mon Jul  5 15:01:01 2021     6               204800       113664

 

Or you can use the GUI to do this, of course, perhaps a little easier. A lot easier when setting maximum usage for an independent fileset. Open the Web GUI, and navigate to the left bar, Files, Filesets, and right click on the new-fileset and select "Edit Quota"

set quota

 

You will be greeted by a warning messages saying: "Quota is Disabled for the File System nas1"
As this is true, let's enable Quota and fileset specific "Disk Free" information:

# mmchfs nas1 -Q yes
# mmchfs nas1 --filesetdf

Reload the web page to freshen the information, and try again:

 edit quota

The GUI said OK, so all is well, right? Let's check that on command line:

# mmlsquota  -j new-fileset nas1 
                         Block Limits                                    |     File Limits
Filesystem type             KB      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
nas1       FILESET     no limits                                   
nas1: Quota management has been enabled but quota accounting information is outdated. Run mmcheckquota to correct and update quota information.

# mmcheckquota nas1
nas1: Start quota check
   1 % complete on Mon Jul  5 15:32:49
...

 100 % complete on Mon Jul  5 15:33:09 2021
Finished scanning the inodes for nas1.
Merging results from scan.
mmcheckquota: Command completed.

# mmlsquota  -j new-fileset --block-size=auto nas1
                         Block Limits                                    |     File Limits
Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
nas1       FILESET           0       500G       500G          0     none |        1       0    0     0      none

# mmlsfileset  nas1 new-fileset -d --block-size=auto
Collecting fileset usage information ...
Filesets in file system 'nas1':
Name                     Status    Path                                               Data
new-fileset              Linked    /nas1/new-fileset                                    4M

 

When exporting the new fileset with SMB, and mounting it on my Mac, the df statement shows the difference between a dependent fileset and a quota enabled independent fileset:

root@scalenode1:~# mmsmb export add new-fileset /nas1/new-fileset/
mmsmb export add: The SMB export was created successfully.

maarten@MacBookPro ~ % df -h -T smbfs
//maarten@192.168.178.199/Documents    1.1Ti   18Gi  1.1Ti     2% 18620414 1208225792    2%   /Volumes/Documents
//maarten@192.168.178.199/new-fileset  500Gi    0Bi  500Gi     0% 18446744073709551614  524288000 3518437222305%   /Volumes/new-fileset

 For some reason the calculation of inode information is a little off, and looks rather strange. Anyway, this concludes the filesystem Tuning and Quota part of the blog, and indeed the blog itself. There are many more subjects that can be discussed, but for building a home NAS I think this should give you a good start. Comment below if you have  a subject you really think should be included​​​​​​​ and I'll consider it. Thanks for reading, and enjoy Spectrum Scale!

​​​​
#Highlights-home
#Highlights
0 comments
200 views

Permalink