File and Object Storage

IBM Spectrum Scale is a cluster file system that supports concurrent access from multiple nodes. Its Cluster Export Services (CES) provide highly available file services to the General Parallel File System (GPFS) cluster via different protocols and Network File System (NFS) is one of them.

This NFS support is based on the Ganesha NFS server which operates in user space. There are multiple commands/tools available for configuring, administering, capturing performance and troubleshooting CES NFS.

A summary of the most commonly used ones is as follows:

Configuration/Administration Related:

This is mostly done via the below GPFS commands:-

1. 
   mmnfs: Helps manage your NFS configuration and all the exports.
   


2. 
   mmces: Administer the CES configuration and check the CES state or events.
   


3. 
   mmauth: Helps handle authentication needs.
   

Troubleshooting Related:

1. 
   Logs: The NFS server writes log messages to the "/var/log/ganesha.log" file.
   
   
   One can decide the amount of logging needed at runtime based on below flexible options:
   
   -- Log level. Eg- select CRIT (critical), INFO (informational), EVENT or DEBUG level messages.
   
   -- Components of interest. Eg- select V4 ACL or NLM locking messages etc.
   
   


2. 
   ganesha_mgr utility: This is a command line utility provided by the Ganesha open source community project.
   
   
   It can be used for displaying all the NFS exports and the current NFS clients. It can help control the logging feature (that was described in the previous section), and to purge various internal caches as well.
   


3. 
   tcpdump: If applications running on NFS clients see any errors, this commonly used tool helps to determine a possible root cause. It can tell you whether the client or server is at fault. As a best practice it is recommended to collect tcpdump from both the client and server side, so that no frames are dropped. And to use the "-w" option for capture, so that tshark can be used to analyze the packets later.
   

Performance Related:

1. 
   ganesha_stats utility: This is another tool provided by the Ganesha open source community project. It helps extract NFS server statistics. It can tell you the statistics based per export. One can see protocol specific details. Eg- V3 or V4 or NLM protocol related. And for filesystem specific operations as well.
   
   It is being enhanced constantly, so that it can capture more details for various different components to identify any potential bottlenecks.
   


2. 
   mmfsadm vfsstats: This is a command that helps extract the GPFS statistics. It tells you the time taken for each of the various calls. Eg- reads, opens, writes, removes etc.
   

Miscellaneous:

1. 
   Administrators may sometimes have to check firewall/port settings, syslog, systemd (especially for any startup issues)
   


2. 
   netstat for any network related problems.
   

Please see below links for more details:

https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1adv_ces_nfssupport.htm

https://ganltc.github.io/data-collection-for-nfs-ganesha-hang-or-performance-issue.html

Ack: Thanks to Malahal Naineni for his review.
#IBMSpectrumStorage
#NFSAccessforSpectrumScale
#Softwaredefinedstorage
#ibmstorage
#IBMSpectrumScale